Featured Post

Building REST APIs using Sails.js (Node.js MVC) and Mongo DB

I have been evaluating various MVC frameworks on Node.js recently. It then led me to Sails.js, which is a MVC framework for building REST A...

Saturday, December 19, 2015

Valuation - Ambuja Cements

Posting the report that we prepared as a group assignment for the course Valuation in IIM Bangalore (October 2015). Hopefully it will help someone else working on a similar assignment.

https://drive.google.com/open?id=0BxJQWtkGpvZFc3hzWHNsMTZqd0U

Strategic Management-Dish TV (DTH Industry in India)

Posting the report that we prepared as a group assignment for the course-Strategy Management as a part of our course in IIM Bangalore (November 2015). Hopefully it will benefit someone else trying to build a similar report in the future.

Sunday, August 30, 2015

Web scraping using Python when pagination is there

In the same context as my last post, where I was exploring web scraping using Python and BeautifulSoup, I hit a road block, where the results were being returned in a paginated form. I dug deeper and found the solution below:


The points that should be noted in the example above are as below:
  • The URL in this case was being changed by sending in the page number in the query string. This is an easy case, since I could change the URL and making it dynamic by passing the page number dynamically.
  • The results were being returned in 11 pages in this case and hence the for loop running from 1 to 11.
  • The rest of the code is fairly self explanatory and if you have any queries, please refer to my earlier blog post on page scraping using Python. If there are still questions, please drop me a note.
I read a note from someone on the social network today:

I think I agree to it. The more I am getting into Python, the more I am enjoying it. 

Saturday, August 22, 2015

Building REST APIs using Sails.js (Node.js MVC) and Mongo DB

I have been evaluating various MVC frameworks on Node.js recently. It then led me to Sails.js, which is a MVC framework for building REST APIs using Node.js. So far whatever I have seen of Sails.js, I have been pretty impressed.

Here I will try to summarize whatever I have learnt since today morning and I will also give an overview of the POC I have built using Mongo DB as the backend database.

What is Sails.js?
Sails.js is an MVC framework for Node.js and from what I have read, it is built with the same pattern as Ruby On Rails. It can be used to build modern enterprise applications from backend to frontend, since it is based on top of Express.js, which is one of the most popular front end frameworks built on top of Node.js. Node.js has already proven itself as a good and scalable server side technology stack, which is performant and quick to learn for anyone with some background in web development using Javascript.

Some of the features of Sails.js:

  1. It is 100% javascript, so easy to learn for someone doing web development
  2. It can be used with most of the popular databases like MySQL, Mongo DB
  3. It comes with BluePrint, using which you can auto generate REST APIs for your backend, which helps in jumpstarting your development
  4. It is front end agnostic, so you can use it with any tech stack you are used to like Angular, Backbone, Knockout etc
  5. It supports Grunt and so all the modules that come with Grunt can be used with your application
  6. It is built on top of Node.js and Express for handling HTTP requests and so has a good solid foundation
How to get started?
I am using IntellJ Idea Ultimate edition V14 for my development with Mongo DB as the backend database. I am connecting to my local instance of Mongo DB using my Mac terminal. Apart from this I am using Postman to send and monitor web requests and test my POC application.

In order to install Sails.js on your mac, use the following:

npm -g install sails

You can create a new project using the following command:

sails new Sailstest

Now navigate to that folder where you created the test project and run the server using the following command:

cd Sailstest
sails lift

Here is what you see:

Your server is now up and if you go to http://localhost:1337 you will see your default Sails app up and running.

You can generate a REST API using the following command:

sails generate api user

This will generate the model and controller for your application-UserController.js and User.js

You can look up your solution structure as below:
With the BluePrint API enabled by default in a Sails.js project, you can immediately go to /user/create and create a new user for your application or go to /user and see a list of all the users in your system. Amazing, isn't it?

Currently, your controller file and model file are blank-we will customize it in a bit. You can disable the above behavior by going to your configuration file /config/blueprints.js and turning it off by setting shortcuts to "false".


So, you can go ahead and generate CRUD REST APIs for your application in a jiffy. If you are doing relational modeling for your application, this is the best way to do it. You can generate all your APIs, host it somewhere and give access to your front end developers to continue to make progress. In the meanwhile, your backend developers can continue to make progress and customize the server side of your application. Fun times!!

Now, let's go ahead and customize our model file - User.js (/api/models/User.js).

I added the following properties for my user entity:
  1. UserName
    1. Type - String
    2. Required - True
  2. Password
    1. Type - String
    2. MinLength - 6
    3. Required - True
  3. Age
    1. Type - Integer
    2. Max - 150 
    3. Required - True
  4. Birthdate
    1. Type - Date
  5. PhoneNumber
    1. Type - String
    2. Default Value - 111-222-3333
  6. Email
    1. Type - email
    2. Required - True
After adding the above into my model file, it looks like below:
Next, I went ahead and added the CRUD functions into my controller file, so that I can control what business logic I want to add to my server code.

My create function looks like below:
req is the HTTP request and res is the HTTP response for your request and next passes control to the next matching route. Here is some context from the web explaining this in more detail:

Express uses middleware functions that have callbacks (functions that get called when an action is completed), and next has that purpose (it's a callback that triggers the next middleware in the Express stack). All Express middleware (that is Connect compatible) have 3 params: request, response, next (which is optional).

For example the static middlware serves static files, the csrf middleware checks for a param when receiving POST requests and the router middleware that handles the routes (what you've pasted above is part of that).

Each middleware can complete it's task and call the next middleware in the queue if some conditions are met (for example the static middleware won't call a next middleware, because it will take care on itself to serve the files, so that the router won't get called).
In the router you don't usually call next because it tends to be the last middleware executed (unless you want something like benchmarking).
If you wanted to create a middleware that denies access to all users that aren't logged in, you would have to call next() only if the user is logged in (so that the next middleware is called, the router in this case and the user can access the page he's looking for), else you would probably redirect him to the login page.
next takes either no parameters at all or an error as a parameter.
Similar to the Create function, I have created the Destroy and Select functions. 

Next I plugged the loop to persist my data into Mongo DB. For this I went and changed the /config/connections.js file. Before this you will have to install Sails-Mongo module using below:

npm -g install sails-mongo
or 
npm install sails-mongo

You can see the change in your package.json file and also see the module added to your node-modules folder.


My connections.js looks like below:

localMongodbServer: {
  adapter: 'sails-mongo',  host: 'localhost',  port: 27017,  // user: 'username',  // password: 'password',  database: 'sailstest'},



My package.json file looks like below:
{
  "name": "sailstest",  "private": true,  "version": "0.0.0",  "description": "a Sails application",  "keywords": [],  "dependencies": {
    "ejs": "~0.8.4",    "grunt": "0.4.2",    "grunt-contrib-clean": "~0.5.0",    "grunt-contrib-coffee": "~0.10.1",    "grunt-contrib-concat": "~0.3.0",    "grunt-contrib-copy": "~0.5.0",    "grunt-contrib-cssmin": "~0.9.0",    "grunt-contrib-jst": "~0.6.0",    "grunt-contrib-less": "0.11.1",    "grunt-contrib-uglify": "~0.4.0",    "grunt-contrib-watch": "~0.5.3",    "grunt-sails-linker": "~0.9.5",    "grunt-sync": "~0.0.4",    "include-all": "~0.1.3",    "rc": "~0.5.0",    "sails": "~0.11.0",    "sails-disk": "~0.10.0",    "sails-mongo": "^0.11.2"  },  "scripts": {
    "debug": "node debug app.js",    "start": "node app.js"  },  "main": "app.js",  "repository": {
    "type": "git",    "url": "git://github.com/sverma4/sailstest.git"  },  "author": "sverma4",  "license": ""}

Then go to your /config/models.js file and enable the connection to your local mongo DB server, as below:
connection: 'localMongodbServer'


Next, I launched my local mongo DB using my terminal and went to http://localhost:1337/user/create. I passed the necessary parameters that my model needs like username, password etc.

My request to the node.js server was as below:

http://localhost:1337/user/create?username=testingsails&password=sailtest&age=55&email=sail.test@gmail.com

You can monitor the connections opening up in Mongo DB and then the request to insert a new user coming to the database:

2015-08-22T17:00:56.547+0530 I WRITE    [conn9] insert sailstest.user query: { username: "sankalp", password: "sankalp", age: 35, email: "sankalp.verma@gmail.com", id: null, phoneNumber: "111-222-3333", createdAt: new Date(1440243056409), updatedAt: new Date(1440243056409), _id: ObjectId('55d85d70f78b25da38ea4fbc') } ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, MMAPV1Journal: { acquireCount: { w: 8 } }, Database: { acquireCount: { w: 1, W: 1 } }, Collection: { acquireCount: { W: 1 } }, Metadata: { acquireCount: { W: 4 } } } 131ms


You will see the following response in your web browser:

Now, you can go and see a list of available users in your Mongo DB in the user collection by going to http://localhost:1337/user. You will see a list as below:

You can go ahead and verify by querying your mongo DB to get a list of the users as well, commands as below:

show dbs

use sailstest


show collections

db.user.find();


Next I am working on plugging this API into my front end. I am building a hybrid mobile app using Apache Cordova. Would be plugging in the API and be generating an Android and IOS app. Have already built an app earlier using this stack, so shouldnt take too long. Watch this space for more.

Feel free to drop me a note and let me know if you have any questions related to Sails.js or Mongo DB.




Wednesday, August 12, 2015

Page scraping using Python

Recently I was trying to figure out how to scrap data out of web pages. The reasons for doing this can be many:

  1. You might be looking for test data for something you are working on
  2. You might be looking for production data for an application that needs similar data and you want to avoid gathering all that data once again
While doing my research, I came across Python getting used for the same. Digging a little deeper, I found a few associated libraries like BeauifulSoup being used for the same.

To provide a little background:
BeautifulSoup: It is a Python library used for screen scraping. Screen scraping essentially means that you "scrap" out data from an existing web page after analyzing the HTML tags on the site. Many tools like fiddler, chrome developer tools can be used to analyze the source HTML of the web page. You will have to analyze the trends, dig to find the div, span tags that might be containing the data you are looking to scrap. It provides parsers like HTML5Lib, LXML and provides an easy way to parse through a page.

In order to install BS4 on mac, use the following:
easy_install BeautifulSoup4

This is much simpler than downloading the installer and installing it. easy_install takes care of downloading, unpacking, installing the package.

Below is my code to scrap data from one of the sites:

          from bs4 import BeautifulSoup
import csv
import urllib2
req = urllib2.Request("http://www.justdial.com/Bangalore/c-%3Cnear%3E-Whitefield/ct-580163", headers={'User-Agent' : "Magic Browser"})

pageFile = urllib2.urlopen(req)

pageHtml = pageFile.read()

pageFile.close()

soup = BeautifulSoup("".join(pageHtml), "html.parser")

f = csv.writer(open("JDWhitefield.csv", "a"))

for innerdiv in soup.findAll("section", {"class": "jrcl"}):
       f.writerow([innerdiv.text])

To analyze the above code:
1. First we import the BeautifulSoup library along with csv and urllib2. 
Urllib2 is a module in Python to open HTML pages and it defines functions and classes to do the same. You can deep dive on urllib2 in the Python docs - https://docs.python.org/2/library/urllib2.html

The function urlopen is provided by urllib2.

2. When requesting the web page using urllib2.request, one can also pass additional information like data to be sent to the server and information like metadata about the request itself, which is sent as HTTP headers. In the example above, I have passed the user agent as Magic Browser. Since we are going to request a web page as a browser would, we have to "fake" a user agent.

3. Then we read the html contents into a variable and close the request. The HTML contents can then be analyzed to find specific tags that might contain the data you are looking for. In the example above, I am looking for a section with class named as jrcl. 

4. Next I am dumping all that data into a csv file using file system operations in Python.

Please note that in some sites the response is in gzipped format. For such sites, you will have to pass a variable in the header that tells that the response is going to be zipped and it should be analyzed appropriately.

Here is an example to handle such requests:

from bs4 import BeautifulSoup
import csv
import urllib2
from cStringIO import StringIO
import gzip

req = urllib2.Request("http://www.lenskart.com/contact-lenses/top-contact-lenses-brands/bausch-lomb-contact-lenses.html", headers={'User-Agent' : "Chrome", 'Accept-encoding':'gzip'})

response = urllib2.urlopen(req)

pageHtml = response.read()
gzipper = gzip.GzipFile(fileobj = StringIO(pageHtml))
data = gzipper.read()
response.close()

soup = BeautifulSoup("".join(data), "html.parser")
f = csv.writer(open("LensKartPrice.csv", "a"))
for innerdiv in soup.findAll("span", {"class": "span6 price-view"}):
f.writerow([innerdiv])




        print innerdiv

Note how I have passed in the header, 'Accept-encoding':'gzip'. I also had to read the contents appropriately due to the compressed nature of the contents. You can check if the contents of a response are coming back as gzipped using a tool like Fiddler or Chrome developer tools.

Feel free to drop me a note in case you have any questions tackling page scraping using Python.

Saturday, May 30, 2015

Few concepts of Organization Structure and Design


Multi-divisional form (also known as M-form or MDF) refers to an organizational structure by which the firm is separated into several semi autonomous units which are guided and controlled by (financial) targets from the center
Multi-divisional forms simply means that there is one parent company, and that parent company owns smaller companies that use its brand and name. The whole organization is ultimately controlled by central management, but most decisions are left to autonomous divisions. Generally, a parent company will own all of the smaller companies and the smaller companies will provide one service. The multidivisional form (M-form) is a particular organizational structure in which a firm is divided into semi-autonomous divisions that have their own unitary structures. The firm is essentially divided into sub-firms, with each sub-firm being responsible for its own production and maximizing its own profit . However, the M-form still has a central office that overlooks the other divisions but the central office’s main responsibility is to develop overall strategies for the business, not to be responsible for each division’s operations. Thus, the M-form proved to be the best strategy for large firms that wished to expand their diversification and appeal to a wider consumer base.
**********************************************************************************************************************
The principal–agent problem (also known as agency dilemma or theory of agency) occurs when one person or entity (the "agent") is able to make decisions on behalf of, or that impact, another person or entity: the "principal". The dilemma exists because sometimes the agent is motivated to act in his own best interests rather than those of the principal. The agent-principal relationship is a useful analytic tool in political science and economics, but may also apply to other areas.
Common examples of this relationship include corporate management (agent) and shareholders (principal), or politicians (agent) and voters (principal).[1] For another example, consider a dental patient (the principal) wondering whether his dentist (the agent) is recommending expensive treatment because it is truly necessary for the patient's dental health, or because it will generate income for the dentist. In fact the problem potentially arises in almost any context where one party is being paid by another to do something, whether informal employment or a negotiated deal such as paying for household jobs or car repairs.
The problem arises where the two parties have different interests and asymmetric information (the agent having more information), such that the principal cannot directly ensure that the agent is always acting in its (the principal's) best interests,[2] particularly when activities that are useful to the principal are costly to the agent, and where elements of what the agent does are costly for the principal to observe.Moral hazard and conflict of interest may arise. Indeed, the principal may be sufficiently concerned at the possibility of being exploited by the agent that he chooses not to enter into a transaction at all, when that deal would have actually been in both parties' best interests: a suboptimal outcome that lowers welfare overall. The deviation from the principal's interest by the agent is called "agency costs".[2]
**********************************************************************************************************************
The prisoner's dilemma is a canonical example of a game analyzed in game theory that shows why two purely "rational" individuals might not cooperate, even if it appears that it is in their best interests[citation needed] to do so. It was originally framed by Merrill Flood and Melvin Dresher working at RAND in 1950. Albert W. Tucker formalized the game with prison sentence rewards and gave it the name "prisoner's dilemma" (Poundstone, 1992), presenting it as follows:
Two members of a criminal gang are arrested and imprisoned. Each prisoner is in solitary confinement with no means of speaking to or exchanging messages with the other. The prosecutors do not have enough evidence to convict the pair on the principal charge. They hope to get both sentenced to a year in prison on a lesser charge. Simultaneously, the prosecutors offer each prisoner a bargain. Each prisoner is given the opportunity either to: betray the other by testifying that the other committed the crime, or to cooperate with the other by remaining silent. Here is the offer:
  • If A and B each betray the other, each of them serves 2 years in prison
  • If A betrays B but B remains silent, A will be set free and B will serve 3 years in prison (and vice versa)
  • If A and B both remain silent, both of them will only serve 1 year in prison (on the lesser charge)
It is implied that the prisoners will have no opportunity to reward or punish their partner other than the prison sentences they get, and that their decision will not affect their reputation in the future. Because betraying a partner offers a greater reward than cooperating with him, all purely rational self-interested prisoners would betray the other, and so the only possible outcome for two purely rational prisoners is for them to betray each other.[1] The interesting part of this result is that pursuing individual reward logically leads both of the prisoners to betray, when they would get a better reward if they both cooperated. In reality, humans display a systematic bias towards cooperative behavior in this and similar games, much more so than predicted by simple models of "rational" self-interested action.[2][3][4][5] A model based on a different kind of rationality, where people forecast how the game would be played if they formed coalitions and then they maximize their forecasts, has been shown to make better predictions of the rate of cooperation in this and similar games given only the payoffs of the game.[6]
In traditional game theory, some very restrictive assumptions on prisoner behaviour are made. It is assumed that both understand the nature of the game, and that despite being members of the same gang, they have no loyalty to each other and will have no opportunity for retribution or reward outside the game. Most importantly, a very narrow interpretation of "rationality" is applied in defining the decision-making strategies of the prisoners. Given these conditions and the payoffs above, prisoner A will betray prisoner B. The game is symmetric, so Prisoner B should act the same way. Since both "rationally" decide to defect, each receives a lower reward than if both were to stay quiet. Traditional game theory results in both players being worse off than if each chose to lessen the sentence of his accomplice at the cost of spending more time in jail himself.
**********************************************************************************************************************
This is a good video explaining the concepts of organizational structure-functional and divisional.


Friday, May 22, 2015

The forces of supply and demand


A market is a group of buyers and sellers of a particular good or service.

Perfectly competitive market meets 2 conditions:
  • The quality of all the goods and services being churned out is the same
  • Not one buyer and seller can control the price of the goods and services, which can be achieved by the virtue of having the number of buyers and sellers above a threshold.

Demand

The quantity demanded is the amount of a good that buyers are willing and able to purchase.

The law of demand states that all things remaining unchanged, the quantity demanded for a good will drop when the price of that good increases.

The demand schedule is a table that shows the relationship between the quantity demanded and the price of a good.

The plot between the quantity demanded and the price of a good is called a demand curve. The demand curve shows what happens to the quantity demanded of a good when its price varies, holding constant all the other variables that influence buyers. When one of these other variables changes, the demand curve shifts. A curve shifts when there is a change in a relevant variable that is not measured on either axis. 


Normal good is one the quantity demanded for which falls when the income falls. 

An inferior good is one the quantity demanded for which increases when the income falls e.g bus rides

Prices of related goods-Goods are called substitutes when the fall in the the price of one good reduces the quantity demanded for the other good e.g. yoghurt and ice cream

On the other hand, goods are called complements when the fall in the price of one good increases the quantity demanded for the other good e.g. petrol and automobiles


Supply

Quantity supplied is the amount of a good that sellers are willing and able to sell.

The law of supply states that if other things remain unchanged or equal, when the price of a good increases, the quantity supplied of that good also increases.

Supply schedule is a table that shows the relationship between the price of a good and the quantity supplied for that good.

The supply curve is a plot between the price of a good and the quantity supplied for that good.








Principles of Micro-Economics-Ten principle of Economics


  • People face trade offs
  • The cost of something is what you give up to get it
  • Rational people think at the margin: rational people make decisions by considering marginal benefits and marginal costs.
  • People respond to incentives - rest is commentary
  • Trade can make everyone better off
  • Markets are usually a good way to organize economic activity
  • Governments can sometimes improve market conditions
  • A country's standard of living depends on its ability to produce goods and services
  • Prices rise when the government prints too much money
  • Society faces a short run trade off between inflation and unemployment


Monday, February 09, 2015

Financial Accounting for Managers

These are my notes from the Financial Accounting class:
  • Assets = Liabilities + Contributed Capital + Revenue - Expenses
Retained Earnings = Revenue - Expenses
Assets = Liabilities + Contributed Capital + Retained Earnings
A = L + C + R - E
A = L + C + R - (E + D)
D = Drawings or total dividends

Retained Earnings = Revenue - (Expenses + Dividends or Drawings)
  • Realization Principle: Revenue is recognized when the goods are delivered or services are rendered. The rationale is that delivery validates a claim against the customer. By implication, if costs are incurred in producing the goods, unless sales are made, such costs are not considered as expenses.
    • There are exceptions to the realization principle: First, in the case of long run construction contracts, often revenue is recognized on the basis of a proportionate or partial completion of the construction. In this case, revenue is recognized without satisfying the test of completion and delivery. Second, in the case of long run installment sales contracts, revenue is recognized in proportion to the actual cash collections.
  • Accrual Principle: The essence of the accrual principle is that net income arises from events that change the owners' equity in a specified period of time and that these are not necessarily the same as the change in the cash position of the business. Thus, realization and accrual lay down the ground rule for the measurement of income.
  • Assets that become expenses:
    • Inventories: These costs are treated as expenses only when the inventory in question  is sold.
    • Prepaid expenses: They become expenses when the services are used or assets are consumed.
    • Long lived assets: Fixed assets have a limited useful life. The costs of such assets expire during the life of the assets in question. Such expiration of the costs of the assets is referred to as "depreciation".
  • A credit sale is recognized at the point of sale, during the accounting period in which the transaction takes place. The uncollected balance at the close of accounting period is reflected as an asset on the balance sheet, regularly mentioned as "sundry debtors" or "accounts receivable".
  • Depreciation: The value of the asset that is of no value or useless towards the end of the life of the asset is called Depreciation Expense. This expense is matched against the revenues of a period to determine the profit.
  • There are 3 main methods to determine the depreciation expense:
    • Straight line method
    • Written down Value
    • Sum of years Digit
  • Straight line method: Here the depreciation is assumed to depreciate equally over the life of the asset. The rate at which the asset will depreciate is agreed upon at the start of the life of the asset. Since the depreciation cost is distributed evenly, it resembles a straight line, if plotted and hence the name of the method.
  • Written down value method: Under this method, the depreciation is higher during the initial years and winds down as the life of the asset increases. Depreciation is taken as a certain rate and it is applied to the written down value of the asset at the beginning of each year. The effect of this method is that the depreciation charge every year, is an amount less than that in the previous year. In effect, larger amounts are depreciated during the initial years.
  • Amortization: This deals with intangible assets like patents, IPRs etc. It is also another cost like depreciation and is matched against revenues. Here we generally use the straight line method and use a shorter period of time to do the calculation.
  • In a profit and loss account, the expenses incurred are on the left hand side and the revenues earned are on the right hand side. In accounting, in the T form, the debits are on the left hand side and credits are on the right hand side. 
  • What are the various parts of a profit and loss account:
    • Sales
    • Sales return and allowance
    • Sales or cash discount
    • Trade discount
    • Other income: includes income arising from incidental activities
    • Cost of goods sold
    • Gross profit: Sales revenue - Cost of goods sold
    • Operating expenses: The cost of the goods sold includes all the costs incurred on producing goods available for sale. The expenses which are not directly related to the company's output on a cause and effect relationship are termed as operating expenses. These are usually grouped under 2 categories:
      • Selling expenses
      • General administrative expenses
    • Personnel expenses
    • Other expenses
    • Operating profit: Gross profit - personnel expenses - depreciation - Other expenses (this is the surplus generated by the company)
    • Interest expense
    • Net profit before tax:is the surplus after meeting all expenses, including interest. This is the profit available to the company as a result of both its operating as well as financing requirements
    • Net Profit or Profit After Taxes: PAT or EAT
  • Flow chart showing Income and Types of Profit
Sales (or Revenues)
Less: Cost of goods sold
           |
Gross Profit
Less: Operating Expenses
          |
Operating Profit
Less: Non Operating Expense
          |
Profit Before Interest and Tax (PBIT)
Less: Interest
          |
Profit Before Tax (PBT)
Less: Tax
          |
Profit After Tax (PAT)
Less: Previous Year's balance of P&L A/C
          |
Profit Available for Distribution
Less: Appropriations
          |
Retained Earnings