ISSN (Print) - 0012-9976 | ISSN (Online) - 2349-8846

A+| A| A-

Reassessing the Impact of SHG Participation with Non-experimental Approaches

This paper critiques recent work that measures the impact of self-help groups, and explains the biases that result from this assessment. Using survey data, it is shown that the methodologies used yield results that misstate the impact. A categorical breakdown is proposed to improve upon these studies, and a simple alternative procedure, the pipeline method, is then estimated to properly correct for selection bias. The results indicate that SHG participation has an impact on assets, livestock income, and salaries. Applying more advanced methods, training is also found to have a positive impact on assets, and empowerment is found to increase with SHG participation.


Reassessing the Impact of SHG Participation with Non-experimental Approaches

Ranjula Bali Swain, Adel Varghese

This paper critiques recent work that measures the impact of self-help groups, and explains the biases that result from this assessment. Using survey data, it is shown that the methodologies used yield results that misstate the impact. A categorical breakdown is proposed to improve upon these studies, and a simple alternative procedure, the pipeline method, is then estimated to properly correct for selection bias. The results indicate that SHG participation has an impact on assets, livestock income, and salaries. Applying more advanced methods, training is also found to have a positive impact on assets, and empowerment is found to increase with SHG participation.

The authors are grateful to an anonymous referee for comments on an earlier version of this paper.

Ranjula Bali Swain ( is at the Department of Economics, Uppsala University and Adel Varghese (avarghese@tamu. edu) is at the Institute for Financial Management and Research, Chennai and Department of Economics, Texas A & M University.

elf-help groups (SHGs) have grown at an astounding pace. Recent data in India finds that their client outreach has reached 45 million households, three times more than the numbers reached by microfinance institutions (MFIs) (Srinivasan 2009:15). With the relatively greater outreach, SHGs may face d iminishing returns and grow slower than MFIs. Still, in 2008-09, SHGs have grown twice as fast as MFIs (7.2% vs 4%) (Srinivasan 2009). Due to the large and growing presence of SHGs in the I ndian microfinance sector and with the government hailing the SHG programme as a crucial poverty alleviation strategy, many want to assess whether SHGs help borrowers. An impact study can find out if SHG membership helps households and provides information to shape the direction of future policy.

Prompted by some of the above reasons, in 2008, the National Council for Applied Economic Research (NCAER) conducted an overarching study on the impact of SHGs (NCAER 2008). Their findings revealed very strong impact of SHG participation on a large number of factors. Unfortunately, their methodology misstates the impact of SHGs. Surprisingly, their pre-post methodology remains similar to the very first impact studies on SHGs. Much of the methodology does not take into account or mention selection biases widely known in the current economics literature.1 Large-scale evaluations such as these by the NCAER have wide impact and reach broad audiences. For example, the recent State of the Sector 2008 Report on Indian microfinance used the NCAER work as “the impact study” on SHGs, without any qualifiers on the methodology (Srinivasan 2009). Thus, it is important to critically reassess the NCAER study in terms of its methods and to provide a more accurate impact study of SHGs.

This paper has two objectives. One, we clearly identify the limitations of pursuing the pre-post methodology that the NCAER and recent impact studies on SHGs follow. Two, we provide easily implementable alternatives for non-experimental data which are more precise in identifying impact. The main alternative, known as the pipeline method, is originally due to Coleman (1999), and has a strong intuitive appeal. It can be implemented directly with proper data collection. The other methods – propensity score matching and structural modelling – are less intuitive, but a software to implement these is easily available. Furthermore, we argue that NCAER’s data set cannot effectively measure impact and provide estimates using a different data set on SHGs. We thus demonstrate the wide differences between the pre-post method and the method we use. We also differentiate among new SHGs, mature SHGs, and non-members. The NCAER paper results are based only on mature SHGs.

march 12, 2011 vol xlvi no 11

Economic & Political Weekly

Using the NCAER method, our results do not reveal much difference between new SHGs and mature SHGs for a large number of impact variables. This indicates that the NCAER results may arise from other observable and unobservable elements that are not fully controlled for in the NCAER analysis. When we control for both observable and unobservable changes, we find a positive impact from participation on assets, livestock income, salaries, education and health expenditure. Surprisingly, we find a negative impact on income. In more advanced impact analysis, we find a positive impact of training on assets. We also find a positive impact on empowerment for SHG members.

On Indian SHGs specifically, impact studies consist of the P uhazendhi and Badataya (2002), commissioned by the National Bank for Agriculture and Rural Development (NABARD) with 115 member respondents from three states. Their methods consist of computing the percentage difference of the means of members’ variables pre- and post-SHG membership. This pre-post methodology is very similar to that of NCAER. Thus, their findings suffer from some of the limitations of the NCAER study. Nevertheless, this Puhazendhi-Badataya study has had much policy influence, and has been quoted by many sources, most recently the Rangarajan Committee on Financial Inclusion (2008). The results find that SHG membership significantly increases the asset structure (30%), savings, annual net income, employment (34%), and s ocial empowerment. The study also finds significant increases in the asset structure, savings (14%), annual net income (6%), and social empowerment.

A more recent report by EDA Rural Systems (2006), jointly with Catholic Relief Services (CRS), CARE, and GTZ2 (hereafter EDA) on 214 SHGs from 108 villages does not attempt a formal impact study but provides summary statistics and interviews with focus groups. The authors disclaim their report as an impact evaluation but still the Rangarajan Committee refers to their work as an i mpact study. Due to inappropriate corrections for selection bias, Tankha (2002:55) comments that, “their findings cannot be considered to be conclusive or even convincing”.

Properly accounting for selection bias entails recognising that the sample of SHG members may differ from the population at large. Thus, one needs to effectively extricate the impact of member characteristics, regardless of SHG participation. For example, if we observe that income for SHG members has grown relative to others, is it because of SHG membership or because SHG members are more entrepreneurial? NCAER may have been motivated by these concerns to measure impact for the same household over time. Unfortunately, NCAER’s method creates problems of its own, which we discuss later in the next section.

In the economic impact evaluation literature, randomisation is the preferred methodology (Duflo et al 2007). However, our data is non-experimental and so we cannot apply experimental methods. As an alternative to randomisation, one can follow exclusion rules to find an instrument that affects credit but not the relevant impact variable. Since the choice of these rules or instruments may differ across authors, differences in opinions can arise. A case in point is the academic debate between Pitt and Khandker (1998), who find impact, and Morduch and Roodman (2009), who find no impact, for microcredit programmes in Bangladesh.

Economic & Political Weekly

march 12, 2011 vol xlvi no 11

Furthermore, SHGs follow no such exclusion rule. Even though SHGs tend to target poorer households, the programme does not follow strict eligibility criteria (as do most microfinance programmes). One can also employ panel data but this suffers from limitations as well. An important objective of this paper is to provide an alternative to randomisation and panel data that is simple for policy use.

The paper proceeds as follows. We first list the limitations of the NCAER study. We then suggest the pipeline method of measuring impact and describe our data set. We then present the results, which apply the NCAER methodology to our data for different categories. We then present and discuss results from the pipeline method. We extend the basic framework to address more advanced impact measurement such as training and empowerment. We end with some policy conclusions, comment on future work, and provide some potential improvements to the current study.

1 Limitations of NCAER Study

The literature refers to the NCAER methodology as pre-post. In 2006, NCAER interviewed SHG members on relevant impact variables such as assets and income. At the same time, field workers asked members to recall information on the same variables b efore they became members. Presumably, by calculating the percentage difference, the authors of the NCAER study were able to attribute the changes solely to the programme, with the individual constant over time.

However, multiple problems arise from this approach. First, the NCAER study is based on recall data from respondents. Members of SHGs were asked during 2006, what the starting position of those variables was before they joined SHGs. (Thus this starting year differs depending on the year of membership.) Clearly, this type of information creates in-built biases in that respondents could exaggerate (or minimise) the impact of SHGs. These biases would be further exacerbated if respondents were to know beforehand that the study’s goal is to measure the impact of SHGs. Inaccurate information will arise and the more mature the group, the further back in time the recall, and the more likely the misinformation.

Second, many other changes occur during the period a household is a member; the impact may result from those changes rather than the programme itself. For example, the government may launch parallel programmes in the area the household r esides in. Then, any measured impact would confound the e ffects of those parallel government programmes with SHG membership. Changes may also occur at the household level, such as any household-level shock. As Karlan and Goldberg (2006:15) a rgue, these pre-post exercises should be viewed as “client monitoring exercises, or client tracking exercises, since while they provide information on how clients’ lives change, they in no way provide insight into the causal impact of the microfinance programme on their lives”. The inadequacy of this approach calls for a control group that resides in an area similar to the household but has not received the benefits of SHG participation.

Third, households have different total years of membership, so the intensity of impact of the SHG is difficult to discern, i e, does the impact come from the last two years or from the first years of membership? From the study, we only know that the average number of years of bank linkage was 5.4, and includes SHG


households with a linkage of more than four years. The selection of which households to include was made randomly so the authors of the study are aware of selection bias but did not take care of the bias on unobservables. The four years takes into account only mature SHGs since as the authors correctly point out, impact may take some time to develop. A common criticism in the impact literature is the presence of “survivorship bias” since no data is collected for dropouts. However, for SHGs, as we will argue in the next section, this problem is not that severe.

Fourth, the calculation by the NCAER is imprecise. Armed with their data, a simple pre-post evaluation would calculate the percentage change in variables at real prices. NCAER though calculates the compound annual growth rate (CAGR). With a base and an end year, the CAGR calculates the yearly return a variable should yield.3 The CAGR is primarily used in finance to “smooth out” the return of assets over time, reducing the volatility of the returns. Comparison should be made over identical time periods, but here as we mentioned, the time periods vary depending on SHG membership date.

Even if the above problems are assumed away, the method of calculating the CAGR is imprecise. We first define the following variables: xi = impact variable for household i in 2006 (year of the sur



x0i = impact variable for household i in year household joined SHG (the year differs across households); T = number of years household is a member; N = number of households.

The NCAER calculates the CAGR in the following manner, taking the means of the relevant variables and raising them to the reciprocal of the mean years of membership:4

1 ¦iT


§ 2006 · N

¨ Ni ¸ 1 ...(1)

¨ x0 ¸

©¦ N ¹

However this method entails a loss of information since it assumes that all information in the data is captured by the mean. A more precise method would calculate the CAGR by denoting it as CAGR-h to differentiate from the other CAGR. It is calculated by the following:


x Ti


¦ xi 1





These methods need not yield the same results, similar to what is well-known with the expectations operator,

ii E(x2006) x2006

z E

E(xi0) xi0

Furthermore, here the second expression is raised to a different exponent depending on the membership time in the SHG. Finding an analytical relationship between the two is difficult and a simulation confirms that the inequality may flow in either direction.5 Still, the CAGR-h provides a more precise measure; with the CAGR one has to assume that all the information is captured by the mean. Due to the limitations of the pre-post method, we now comment on methodologies which can improve measurement of impact.


2 Methods and Data

In this section, we outline the basic conditions for an effective impact study, provide an alternative methodology to randomisation, and summarise our data on SHGs. As we argued in the previous section, a pre-post methodology will not adequately account for the changes that SHG member households face. Identifying a comparison or control group which has not received the benefits of SHG membership but which has experienced the same changes as the treatment group (SHG members) will help in this regard.

Assume one has both baseline (beginning period) and current data for the control and treatment groups and that both are from the same area. One can then compute the percentage difference in relevant impact variables for members vis-á-vis the control group. If the percentage change for the treatment group’s variable is significantly greater than the comparison group, then that suggests the programme had impact.

In the ideal scenario, with panel data, one can collect the data yearly and compute these differences. However, like randomisation, panel data faces limitations when applied to SHGs. First, SHG programmes are already widespread, reaching out to 45 million households. One would like to compute the impact of participation for those households relative to previous loans. Second, panel data costs are large and by the time researchers collect and process the data, the programme may have moved on. Third, providing a control group without services for say, one year, is perhaps justifiable, but one cannot deny access and maintain a control group for long. If one wants to measure the impact of a long-term variable such as education, the task is more difficult because one has to hold a control group for much longer. So far, no one has collected panel data for SHGs.

As an alternative to panel data, we can proceed in a different direction. So far we have argued that for a relevant impact study, one would need a control group. Furthermore, ideally one would need to choose a control group with the same external conditions as the treatment group. This selection raises other problems. If one selected both treatment and control groups from the same village, then one would wonder why some households chose or chose not to join SHGs. In other words, SHG members may be special. Regression analysis allows one to control for any observable differences such as age and gender. With cross-sectional data, however, one cannot control for unobservable differences such as entrepreneurial skill and talent.6

A design feature of SHGs provides us with an opening to properly measure impact. By design, members have to wait to receive a loan from the bank (about six months). We can exploit this f eature to identify self-selected SHG members who have not yet r eceived a loan. In particular, our data (which we describe in more detail below) allows the following. In certain districts, some members are currently active members of SHGs. In other villages, in these same districts, members from newly formed SHGs have been selected but have not yet received financial s ervices. Thus, the control group in our sample consists of the new SHGs, while mature SHGs form our treatment group.7 We hypothesise that the old and new SHGs have similar unobservables. We also have information on non-members from these districts so that we can condition on the selection, i e, joining the SHG. In other words, we have two control groups. The first control group includes SHG members who have

march 12, 2011 vol xlvi no 11

Economic Political Weekly

not yet received loans while the second includes non-members who have not received loans and are not members.

Raising the level of aggregation to another level, such as d istricts (where both mature and new SHGs reside) helps hold district-specific conditions constant. Similarly, NABARD’s decision to expand the SHG programme occurs at the district level, without any specific policy targeting certain villages over others.8 This choice of district as the level of aggregation, appropriate for SHGs, differs from most research, which uses villages as the a ggregation unit. To account for the remaining village-level varia bility, we employ village-level characteristics. To the extent that district-wide effects may spill over from mature to new members and nonmembers, the estimates here would underestimate that impact.

One critical check is whether old and new villages differ substantially.9 In other words, did policymakers prefer expanding the SHG programme in certain villages first over others? In our

Table 1: Logit Estimates of Placement discussions with various officials, of SHG Programmes

and our reading of various

Village-level Variables SHG placement

N ABARD and Reserve Bank of

(in km unless noted)

Distance block -2.19 (0.98) India (RBI) documents, we did

Distance haat -0.45 (0.05) not find that certain villages

Distance paved road -11.55 (0.98) were preferred over others. As

Distance bank 1.19 (0.37) an additional check, we can Distance market 7.02 (0. 78)

check the observables at various

Distance healthcare -0.156 (0.02)

levels of aggregation. For the

Distance bus stop 11.30 (1.08)

village level, we first present

Male wage (Rs) 1.78 (1.30)

evidence in Table 1 which esti-

Female wage (Rs) -2.96 (1.44) *** Significant at the 1% level. ** Significant mates a logit regression for old at the 5% level. * Significant at the 10% level. All regressions include district fixed effects.

and new SHGs at that level. Note

Analysis based on 220 observations. Absolute

that none of the village-level

t-ratios in parentheses computed with White heteroskedasticity-consistent standard errors. variables are significant.10

What about non-governmental organisations (NGOs) – do they favour certain types of villages for linkages earlier than others? First, NGOs operate within villages without anticipation of a linkage, i e, they move independently of SHG linkage in their own development work. Second, in comparing linkage models (since some groups are bank-formed and some are NGO-formed), we do not find a discernible difference in linkage choice of villages with old and new SHG members.11 Thus, we do not find any discernible evidence of NGOs favouring certain villages first over others. Table 2: A Comparison of the Observables Finally, to check for differ

between New and Old SHG Households

ences in the observable charac

Household-level Variables Coefficient for Ȗ

teristics for old and new SHGs at

Age -1.94 (1.39)

the household level, we ran re-

Gender -0.001 (0.07)

gressions of the following type:

Dependency ratio -0.01 (0.32)

Number of males 0.07 (0.36) Xijs = ĮD + ȕMijs + ȖTijs ...(3)


Primary education -0.08 (1.11) Secondary education 0.06 (0.95) where Xijs is the observable Tertiary education -0.01 (0.40) characteristic, D is a vector of

sLand owned in 2000 0.09 (0.45) district dummies, Mijs is a mem-

Shock dummy 0.03 (0.47)

ber dummy which takes a value

Land cultivated in 2003 0.06 (0.20)

one for members and zero other

*** Significant at the 1% level. ** Significant at the 5% level. * Significant at the 10% level. For the wise, Tijs is a treatment variable full regression equation, see Equation (3) in the

which takes on the value one

text. Results of other coefficients are suppressed. Analysis based on 840 observations. Absolute

for old SHGs and zero for new

t-ratios in parentheses computed with White heteroskedasticity-consistent standard errors. SHGs. Thus, the s ignificance of y

Economic Political Weekly

march 12, 2011 vol xlvi no 11

savings and lending). Household characteristics include age, gender, dependency ratio, education dummies, and a shock dummy.13 In order to control for initial wealth, we employ land owned three years ago. For village characteristics, in addition to male wage, we include the following distance variables

– paved road, market, primary healthcare indicates any difference over and beyond district and self-selection differences. The results in Table 2 indicate that none of the household-level variables were significant. Thus, at the village, NGO, and household level, we do not find any differences at least at the observable level between old and new SHG members.

This study focuses on a diverse set of ten representative districts from five states.12 The data used for the empirical analysis in this paper was collected from 1,000 households in 2003. Additionally, recall data for the year 2000 was also collected. Within the states, we avoided districts with over- and under-exposure of SHGs and selected SHGs for good operational links with banks.

For this particular study, the collected data was further refined. Of all the respondents, 114 were from villages with no SHGs. Since these households were not provided the opportunity to self-select, these were dropped. Sixty old and new SHG respondents were from the same village and this would contaminate the sample since the earlier signees may have different observable and unobservable characteristics compared to later signees. Of the remaining sample, 604 respondents were from mature SHGs, 185 from new SHGs, and 51 were non-members.

For the critical variable in our test, SHGMON, or the number of months since a member had joined a SHG, we made the following adaptations. Since an SHG is bank-linked only six months after formation, we needed to take those six months into account. A lmost all the new SHG respondents in our data had been members for less than six months, and for these SHGMON was coded as zero. Only 14 of these new respondents had been members for more than six months, so we subtracted six months from the date of formation. Similarly, for the mature SHGs, respondents’ SHGMON was calculated by subtracting six months from their membership. A few mature SHG respondents (46) did not report the date of their SHG formation. For these households, we used the number of the months since they received the first SHG loan for SHGMON.

As suggested by Doss et al (2008), we divide assets into six categories – land owned, livestock wealth, dwelling and ponds, productive assets, physical assets, and financial assets (including

Table 3: Descriptive Statistics for Selected Variables (in Rs)

Variable Mean Standard Deviation

N 840

Total assets 1,09,034 1,43,813
Consumer durable assets 1,439 3,011
Income 16,377 16,584
Agricultural income 15,544 16,413
Livestock income 205 662
Wages 4,660 10,943
Salaries 312 943
Self-employed income 188 1,253
Other income 128 558
Food expenditure 1,225 986
Non-food expenditure 1,878 9,313
Expenditure on education 202 545
Health expenditure 238 746
Annual savings of household 657 4,564

centre, and bus stop, also shown in Table 1. Table 3 lists selected descriptive statistics.

Ideally, one should also collect data on dropouts to estimate the “survivorship bias”. However, the dropout rate for SHGs is not


severe – the EDA study estimated the dropout rate as 9.8%, below of a group similar to that of the NCAER, which would lead one to
the 20-30% cited by Aghion and Morduch (2005) and Karlan (2001) conclude that membership has a strong positive impact for a
as a severe problem.14 Furthermore, the EDA study indicates that large number of variables.
almost 50% of the SHGs had no dropouts, and one-third had two or However, breaking down by categories provides an additional
fewer dropouts. The very poor had a higher dropout rate of 11% insight. First, by comparing mature SHGs to new SHGs and assum
but not considerably higher than the 7% of the non-poor. A more ing all other changes are constant, one can assess what additional
systematic analysis by Baland et al (2008) also did not find much advantage participation provides. The categories with distinct ad
attrition in SHG groups but did note that those who did drop out vantages are salaries (people with permanent or regular work that
were more socially disadvantaged. We did not track the dropouts receive a fixed monthly amount), self-employed i ncome, and ex
but considering the slightly higher dropout rate of the very poor in penditure on non-food items.16 The categories with distinct disad-
SHG programmes, the estimates we present will slightly overesti vantages are livestock income, wages and health expenditure.
mate the impact. Thus, the results of this study are conditional on These last results indicate that SHG participation may not provide
the remaining mature SHG members being similar to the dropouts. any added benefits in that future members were already more suc-
Keeping in mind the outlined procedure and data , we can esti cessful than current members in these cate gories. For the rest of
mate the following regression: the variables, SHG participation does not have much impact.
Iijs = Į+ ĮXijs + ȕVjs + ȜDs + ȖMijs + įSHGMONijs +Șijs ...(4) Comparing new SHG members to non-members indicates the
advantage of the self-selection since new SHGs have not received
where Iijs is the impact, measured in terms of asset ac cumulation or any of the returns from participation yet. These advantages hold
income generation, for household i in village j and district s; Xijs are for assets, income, wages, self-employed income, and savings.
the household characteristics; Vjs is a vector of village -level charac- New members are at a disadvantage with regard to non-food
teristics; and Ds is a vector of district dummies that control for any e xpenditure and health expenditure. Presumably, the asset
district-level difference. Here, Mijs is the membership dummy vari accumulation is triggered by financial discipline since savings are
able, which controls for the selection bias. It takes the value one for required before households can link with a bank and receive a
both mature and new SHGs. It takes the value of zero for those loan. We avoid comparing mature SHG members to non-members
villagers that have chosen not to access the programme. Here, since that would conflate participation and self-selection.
SHGMONijs is the number of months that SHG cre dit has been As we illustrate, we can improve upon the pre-post methodo
available to mature members, and is exogenous to the households. logy that the NCAER study employs with the breakdown by
groups. Still, these CAGR calculations rely on recall data, prone to
3 Results of the CAGR Method inaccuracies. Furthermore, a hidden assumption is that all
In order to explicitly examine the methodological issues, we esti changes and variables, both observable and unobservable, are
mate impact using different approaches for our data. Table 4 the same for all groups. The alternative methodology that we
presents results from the NCAER CAGR method with our data. These calculations differ from the NCAER because h ere the e mploy can improve on these limitations.
lations of the base variable are all from three years ago. On the calcu 4 Results of the Pipeline Method
other hand, the NCAER calculates the base variable from the year We now provide results from the pipeline method. There are three
when members joined SHGs. Both calculations depend on recall main differences between these estimates and those from CAGR.
data for the base variable. One, we use non-members and new SHG members as controls.
We differ from the NCAER in providing three di stinct groups This takes into account region-wide impacts and controls for
for the results.15 The three categories represent mature SHGs, member self-selection. Two, we use conditioning variables such as
new SHGs (with membership but no loans), and non-members (no age, education, and land owned three years ago. These take into
membership and no loans). The first column presents the results account differences due to other individual and village-level varia-
Table 4: NCAER-Style CAGR tion. For example, the CAGR method may find a positive impact on
Variable Name/Test Mature SHGs New SHGs Non-Members savings not necessarily because of membership but because dur
(1) (2) Total assets 8.2 8.2 Total savings 9.8 12 Income 8.1 8.3 (3) 2.4 -23.7 5.9 ing the time of participation, SHG members have reached the age when households begin saving. Three, we employ only the crosssectional data from 2003, without relying on recall data for 2000.
Agricultural income 8.1 8.5 5.9 Table 5 (p 55) presents the results of the pipeline method for as-
Livestock income 5.6 24.1 12.1 sets and income. Note that members begin with fewer assets than
Wages 1.9 4.7 1.5 non-members but membership provides an avenue for asset a ccu-
Salaries 9.3 -2.6 -1.6 mulation (as seen through SHGMON). Column (2) indicates that if
Self-employed income 21.1 -4.0 -41.0 one were to suppress the membership dummy, in other words not
Other income -1.0 1.6 Food expenditure 9.8 8.9 Non-food expenditure 18.5 11.5 Expenditure on education 16.7 15.1 23.5 11.7 45.9 19.3 account for member self-selection, one would erroneously conclude that membership has no impact on assets, understating impact. Column (3) shows that as SHG members become more mature, their
Health expenditure 11.3 15.4 29.1 current income goes down. Thus, there arises a trade-off between
NCAER-style CAGR = (variable mean in 2003/variable mean 2000)1/3-1. current income and the transition to future asset accumulation.
54 march 12, 2011 vol xlvi no 11 Economic Political Weekly
Table 5: Pipeline Method Results for Assets and Income (in thousands)
Total Assets Total Assets Income
(1) (2) (3)
Member -42.5 (2.11)** 3.59 (1.40)
SHGMON 0.65 (1.99)** 0.46 (1.40) -0.07 (1.75)*
Age (years) 0.09 (0.16) 0.16 (0.28) 0.13 (1.82)*
Gender (female=1) 7.91 (0.60) 9.95 (0.75) -0.33 (0.13)
Dependency ratio 41.43 (2.23)** 38.64 (2.14)** -10.65 (4.01)***
Primary education 22.64 (1.89)* 23.74 (1.93)* -1.85 (1.17)
Secondary education 31.84 (2.65)*** 31.23 (2.61)** -3.38 (1.99)*
College education 47.70 (1.85)* 47.04 (1.83)* -6.04 (1.80)*
Land three years ago (acres) 44.20 (8.41)*** 43.84 (8.41)*** 1.73 (4.31)***
Average shock -0.12 (0.01) -0.21 (0.02) 2.00 (1.50)
Distance paved road (km) -7.30 (2.35)** -7.62 (2.44)** -0.22 (0.56)
Distance bank (km) 0.84 (0.76) 0.81 (0.72) -0.13 (1.01)
Distance market (km) -1.66 (1.58) -1.81 (1.72)* -0.05 (0.31)
Distance healthcare (km) 2.39 (1.00) 2.78 (1.19) -0.06 (0.25)
Distance bus stop (km) 4.16 (1.32) 4.49 (1.42) 0.02 (0.06)
Male wage (Rs) -0.47 (1.01) -0.36 (0.79) 0.001 (0.02)

*** Significant at the 1% level. ** Significant at the 5% level. * Significant at the 10% level.

All regressions include district dummies. Analysis based on 840 observations. Absolute t-ratios in parentheses computed with White heteroskedasticity-consistent standard errors clustered by village. Income is a Tobit regression with non-White standard errors. See text for definitions of variables.

Tables 6 and 7 present the results for other impact variables.17 The regression is identical to the previous pipeline method but we focus on the variables of interest, the member dummy and months of membership (SHGMON). The results indicate a movement away from agricultural and wage income to livestock and salary income. For livestock income, we see an increase for members, who actually begin poorer but increase their livestock i ncome over time. The food and non-food expenditure results witness no change. Presumably, food and non-food expenditures have increased for others as well. Other interesting results show that both education and health expenditure increase with member ship. In particular, members begin with lower expenditures on health vis-á-vis non-members. Table 6: Tobit Estimates of Impact on Select Disaggregated Income (in thousands)

Agriculture Income Livestock Wage Salary Self-Employed Other Income
Member 2.74 (0.99) -0.53 (1.75)* -3.71 (0.73) 1.48 (1.05) 2.72 (2.25)** -0.22 (0.58)
SHGMON -0.06 (1.45) 0.01 (3.14)*** -0.14 (1.76)* 0.03 (2.19)** -0.01(0.39) 0.01(0.82)

*** Significant at the 1% level. ** Significant at the 5% level. * Significant at the 10% level. All regressions include the right-hand side variables of Table 8 and district dummies. Analysis based on 840 observations. Absolute t-ratios in parentheses. See text for definitions of variables.

Table 7: Tobit Estimates of Impact on Expenditure Variables and Savings (in thousands)

Food Non-Food Education Health Savings

Member 0.12 (0.61) -1.10 (0.46) -0.15 (1.32) -0.36 (2.90)*** -1.04 (0.59)

SHGMON 0.001 (0.35) 0.03 (1.37) 0.004 (2.44)** 0.003 (1.79)* .046 (1.91)*

*** Significant at the 1% level. ** Significant at the 5% level. * Significant at the 10% level. All regressions include the right hand side variables of Table 8 and district dummies. Absolute t-ratios in parentheses. See text for definitions of variables.

To obtain figures comparable to Table 4 from the pipeline method, we find the marginal impact from the regression equation and multiply by the mean months of membership (26) for mature SHGs.18 This will provide us with the total impact from SHG participation. We then divide by the base value in 2003 to find the percentage increase. Column (1) in Table 8 provides calculations from this approach. In Column (2), we use the same e stimates as in Column (1) but use a CAGR calculation. We can compare the results from Column (2) to those calculated in the previous section, listed again in Column (3). Surprisingly, for a ssets we find a very similar result, a return of roughly 8% from

Economic Political Weekly

march 12, 2011 vol xlvi no 11

Table 8: Comparison of Impact Estimates (%)
Variable/Method Pipeline CAGR from Pipeline CAGR from Our Data CAGR from NCAER
(1) (2) (3) (4)
Physical savings NA NA NA 14
Total assets 19.5 8.5 8.2 NA
Financial savings 51.9 21.2 9.8 14.3
Income -13.7 NA 8.1 6.1
Agricultural income 0 0 8.1 5.5
Livestock income 129 46.3 5.6 11.2
Wages -86.2 NA 1.9 5.3
Salaries 263 80.8 9.3 5.4
Self-employed income 0 0 21.1 7.0
Other income 0 0 -1.0 7.3
Food expenditure 0 0 9.8 5.1
Non-food expenditure 0 0 18.5 5.4
Education expenditure 82 31.7 16.7 5.6
Health expenditure 15.6 6.7 11.3 5.5

Column (1) calculated by the coefficient on SHGMON in Tables 3-5, multiplied by the mean months for mature SHGs (26) and divided by the base in year 2003. Column (2) calculated in a way similar to Column (1) but using the formula CAGR= ((variable in 2003+26*coefficient on SHGMON)/variable in 2003)1/2.2-1, Column (3) is extracted from Table 2, column (1), whereas column (4) is from NCAER (2008), Chapter 5.

membership. In Column (4), we transpose the results from the NCAER study. Unfortunately, NCAER does not provide results for total assets but only for physical and financial assets.

For the rest, we find mixed results when compared to NCAER. For a number of variables, such as financial savings, livestock i ncome, salaries, and expenditure on education, we find a much stronger impact using the pipeline method compared to the NCAER and our previous CAGR calculations. For a large number of variables, including agricultural income, self-employed income, other income, food and non-food expenditure, the pipeline method yields no statistically significant impact. For income, as mentioned earlier, we surprisingly find a negative effect and thus we cannot calculate CAGR from the pipeline method. This contrasts sharply with the positive 6% increase found by NCAER. Thus, the difference is not due to membership but due to the other covariates. For example, SHG members may be younger or have a lower dependency ratio, resulting in self-selection and higher income.

5 More Advanced Impact Evaluation Techniques

The pipeline method outlined in the previous section can be combined with other methods to address other relevant impact questions. In particular, SHGs provide skill formation training for their members. For training, NCAER (2008) does not provide any CAGR calculations but simply states that about half of the members receive no training. NCAER (2008) further states that only 30% of the SHG members receive adequate training and that most of the training need is technical.

One can investigate whether training positively affects members. In this case, we would face a double selection problem, at one level the membership issue and at the second level, whether members choose to undertake training or not. Due to reasons similar to membership selection, for training impact on SHGs, one would need to turn to non-experimental methods. Since much of the choice of training is up to the individual, we need to account for this selection problem.

A popular non-experimental method is propensity score matching. Intuitively, we can first predict (either through a logit


or probit equation) who will decide to train, and generate a propensity score or a probability for choosing to train. We then can match these probabilities with those who decide not to train and any differential between the treatment and control groups we can attribute to training. We combine these with our previous equation, equation (3) which accounts for member selection. The combination of these methods is referred to as regressionadjusted matching. These methods can be implemented directly with readily available software programmes (Ichino and Becker 2002, Leuven and Sianesi 2009).

By employing regression-adjusted matching, one can actually test the impact of training on outcome variables. Table 9 presents the results of this method using one particular algorithm, the LLR or local linear regression method. Our results indicate that training positively impacts assets but has no effect on income, consonant with our previous results in Table 5, which focused on participation alone. Table 9: Regression-Adjusted Matching Estimates (LLR Algorithm) of Training Impact on Assets and Income

Bandwidth Gross Assets (Regression-Adjusted) Income (Regression-Adjusted)
(1) (2)
Bandwidth 1 2.01** (1.99) .008 (0.60)
Bandwidth 4 2.01** (2.12) .008 (0.64)

** Significant at the 5% level. * Significant at the 10% level. P-values in parentheses. Figures in parentheses are standard errors created by bootstrap replications of 200. Covariates of regression are the same as Table 3. See text for definitions of variables.

A similar issue arises with measuring the impact of variables such as empowerment. Here NCAER (2008) includes a table ( Table 5.29 specifically) on the impact of social empowerment of women. They ask women whether they feel more empowered a fter joining the SHGs. Not surprisingly, 92% claim that they feel more empowered. Other than the many problematic issues outlined in the previous sections, here the study draws its information from self-reporting households, many presumably knowing that they are asked the information for studying the impact of SHGs.

In Table 10, we report a similar exercise with our data as that done by NCAER, though with an additional category breakdown of mature and new SHGs and non-members. Note that for a number of categories such as increase in self-confidence, mature and new SHG members yield virtually identical results. Since new members have not been members for long, either the act of joining empowers members or respondents believe they have to r espond positively when asked this question. For many of the decision-making variables, we do not observe much difference between the mature groups and non-members. These differences may indicate that in terms of decision-making, membership may not matter.

Table 10: Social Impact Variables by Participation (%)

Variable Mature New Non
SHGs SHGs Members
N 604 185 51
Numbers that report:
Taking crucial decisions in purchase of raw materials, and
pricing of the product 64 48 51
Planning your own activities 50 48 37
Creating any additional job opportunity for others 25 27 10
Able to arrange the credit and other inputs in time 64 52 16
Self-confidence has increased 89 88 26
Treatment of spouse has become more respectful 49 49 8
Confident of meeting financial crisis 88 84 31
Average number of officials, bankers you have met 1.34 1.05 0.41
Good communication skills 73 73 24
Decision-making in family planning decisions 24 19 24
Decision-making in children’s marriage 14 11 28
Decision-making in buying and selling property 19 17 18
Decision-making in sending daughters to school 31 25 31
Decision-making in family matters 47 39 37
Increased access to sanitation 34 34 33

Apart from being a conceptually complex issue, women’s empowerment is a latent variable and cannot be measured directly. An effort to measure empowerment such as by using a composite index suffers from the arbitrary assignment of weights to variables. An additional difficulty is that many of the variables are

o rdinal in nature and assigning a numerical value may be inappropriate. One possibility is to follow Jöreskog’s LISREL structural modelling method in which the unobserved univariate continuous distribution generates an observed ordinal distribution as a latent response distribution (Jöreskog 2002). The underlying variable therefore assigns a metric to the ordinal variable. In contrast to the pipeline method, Bali Swain and Wallentin (2009) divide the groups into SHG members and control group. They employ the following structural equation for analysing the change in women’s empowerment between 2000 and 2003:

Ș= Į+ Ƚȟ + Ȣ ...(5)

where Ș is the latent women's empowerment of the respondent at year 2003 and ȟ denotes the latent women's empowerment in 2000. The symbol Ȣ indicates the error in the structural equation. However, the estimate of prime interest is Į, the mean change in the latent women's empowerment from 2000 to 2003. The results from the general structural model confirm that the mean difference is highly significant for the programme group but statistically non-significant for the control group. This indicates a s ignificant increase in the level of women’s empowerment over time for the SHG members but no such change for the members of the control group. We need to state two caveats for these results. One, we need to combine LISREL with self-selection correction and two, the 2000 variables were obtained through recall. Still, one can presumably use these variables again with readily available software to better measure the impact of empowerment.

6 Conclusions

We briefly summarise and provide some policy conclusions. In this paper, we have first evaluated the recent NCAER impact study on SHGs, which follows a methodology similar to other impact studies on SHGs. We have argued that the methodology, based on recall data with lack of a proper control group, can lead to misleading impact measurement with the CAGR. Using different data on SHGs, we calculate the CAGR for our data. We show how the CAGR methodology can be improved if these calculations are also made for control groups such as new SHG members and nonmembers. We then show how the pipeline methodology can be readily implemented for evaluation. In our results, we find a strong impact on total assets and an increase in livestock income and salaries. Marginally, both health and education expenditure are affected. We find no impact on other variables.

march 12, 2011 vol xlvi no 11

Economic Political Weekly

We then proceed to extend the methodology to two different findings should qualify the methodologies used in the impact cases: double self-selection and latent variables. For double studies, so that readers may gauge their weight. Some improveself-selection, as in the case of training, we use regression-ad-ments of the results here would lie in collecting a larger sample, justed propensity matching to find a positive impact of training especially on non-members. Panel data collection for SHGs would on assets. For the latent variables, we use the structural equa-also be useful, though the costs of collecting and processing the tions model to find a positive effect of empowerment on SHG data are high, especially when decisions on SHGs need to be made members vis-á-vis a control group. In general, SHG membership in real time. In terms of policy, proper impact evaluation is critihelps asset accumulation (a long-term effect) but has potentially cal. As the authors of “When will we ever learn?” (Centre for negative effects on current income generation. G lobal Development 2006) argue, a gap occurs in impact evalua-

Future studies on the impact of SHGs could use these tech-tion that needs to be closed so that impact studies can be properly niques. More importantly, policy reports that mention impact addressed and evaluated.

Notes regional balance – two southern states, one west-Coleman, Brett (1999): “The Impact of Group Lending 1 For example, the widely used textbook by Aghion

ern, one eastern, and one central. For example, in Northeast Thailand”, Journal of Development and Morduch, The Economics of Microfinance,

Karnataka was dropped for fear of over-represen-Economics, 60(1): 105-41. published in 2005, devotes an entire chapter to tation of southern states where SHGs are promi-Doss, Cheryl, Caren Grown and Deere Carmen Diana selection issues.

nent. In an interesting coincidence, NCAER chose (2008): “Gender and Asset Ownership: A Guide to very similar states. The sponsor of the study was Collecting Individual-level Data”, Policy Research

2 Now known as Deutsche Gesellschaft für Inter-

Swedish International Development Corporation Working Paper, World Bank, Washington DC.

nationale Zusammenarbeit (GIZ). Agency (Sida). Duflo, Esther, Rachel Glennerster and Michael

3 Thus, the CAGR is undefined when base is greater 13 The shock dummy is coded as one if the respondent K remer (2007): “Using Randomisation in Ecothan the end and when the base is zero. These resreports yes to any one of the following – social nomics Research: A Toolkit” in T Schultz, Paul trictions are problematic for economic impact and religious emergency, failure of crops (includ-and J Strauss (ed.), Handbook of Development variables, less so for financial variables.

ing failure due to lack of rain), illness in family, E conomics, Volume 4 (Oxford: North Holland).

4 Note that the NCAER method differs from the loss of work for one of the earning members or a EDA Rural Systems (2006): Self Help Groups in India: traditional CAGR calculation because it uses natural catastrophe (like drought, cyclone or A Study of the Lights and Shades (Gurgaon: EDA mean years of membership (number of years of floods). This information was asked for both Rural Systems).

membership differ across households). In calcu2000 and 2003. We averaged the two to create an Ichino, Andrea and Sascha Becker (2002): “Estimalating the CAGR traditionally, the base and end average shock variable. tion of Average Treatment Effects Based on Proyears are the same.

14 The dropout issue is twofold (Karlan 2001). First, pensity Score”, Stata Journal, 2(4): 358-77.

5 Results of the simulation run in programme R are

due to the incomplete sample bias, dropouts are Jöreskog, Karl (2002): Structural Equation Modelling

available from the authors. Actually in calculating impacted differently such that an impact assess-with Ordinal Variables Using LISREL (Chicago: the CAGR-h, one runs into a number of problems ment does not take into account the whole pro-Scientific Software International).

such as undefined zeros and possible negative gramme, only better performers. In the second Karlan, Dean (2001): “Microfinance Impact Assessnumbers.

type, the attrition bias, the active borrowers are ments: The Perils of using New Members as a Con6 With panel data, assuming that these unobsernot either failed borrowers or the stars that chose trol Group”, Journal of Microfinance, 3(2): 76-85.

vable differences are fixed and linear over time, to graduate. If the failures are more likely to drop Karlan, Dean and Nathanael Goldberg (2006): “The one can eliminate the difference. But as argued out, comparing old and new borrowers overesti-Impact of Microfinance: A Review of Methodoabove, one faces difficulties with using panel data mates impacts. logical Issues”, Doing Impact Evaluation 7, World for SHGs.

15 We follow NCAER in including all the zeros from Bank Thematic Group on Poverty Analysis, Moni7 One caveat of this approach is that we need to the variables. Thus, this captures the mean effect toring and Impact Evaluation, Washington DC.

assume that the behaviour of the new SHG memon the categories. Not including zeros would re-Leuven, Edwin and Barbara Sianesi (2009): “PSbers has not changed while awaiting loans. Howsult in markedly different results. For example, MATCH2: Stata Module to Perform Full Mahaever, an advantage of the slow incubation period for a category such as livestock, one could obtain lanobis and Propensity Score Matching, Common of SHGs is that members know the nature of the for mature, new, and non-members, the following Support Graphing, and Covariate Imbalance Testwait for some time and will not change their CAGRs: -1, 30, and 62. ing”, Department of Economics Statistical Softb ehaviour radically as might happen with a one16 Note that we cannot apply a t-test because the ware Component s432001, Boston College.

time infusion.

CAGR yields only means without distributions. Morduch, Jonathan and David Roodman (2009): “The 8 NABARD’s, or the bank’s, decision to form a link-

Another advantage of using CAGR-h is that it will Impact of Microcredit on the Poor in Bangladesh: age programme might follow a non-governmental

yield a distribution. Revisiting the Evidence”, Working Paper, CGD, organisation’s (NGO) choice. We do not have

Washington DC. i nformation on whether NGOs favour certain

17 Note that for our purposes, we simply run a

r egression for all values. In this case, this is a Pitt, Mark and Shahidur Khandker (1998): “The v illages over others within certain districts.

standard Tobit that takes into account the censor-Impact of Group-Based Credit Programmes on 9 We thank an anonymous referee for suggestions

ing of the variables. We include all the zeros in Poor Households in Bangladesh: Does the Gender on these arguments. these regressions. of Participants Matter?”, Journal of Political 10 We also ran this regression for village-level vari-Economy, 106(5): 958-96.

18 We need to assume that marginal and average im

ables that we used for our eventual impact regres-NCAER (2008): “Impact and Sustainability of SHG

pact are equated and rates of return on membersion, and found virtually the same results. Bank Linkage Programme”, Report submitted to

ship are constant. 11 These are the three kinds of linkages under the SHG GTZ and NABARD, NCAER, New Delhi. programme. In linkage 1, banks form and fi nance Puhazendhi, V and K C Badataya (2002): “SHG-Bank groups. In linkage 2, NGOs form groups and banks Linkage Programme for Rural Poor – An Impact


finance them. In linkage three, banks provide Assessment”, Paper presented at the Seminar on financing to the NGOs who then lend to the groups. SHG-bank Linkage Programme, New Delhi, 25-26

Aghion, Beatriz and Jonathan Morduch (2005): The (NGOs also form the groups in this type of linkage.) November.

Economics of Microfinance (Cambridge, Massa-For the (new) old SHGs, the proportions for the three Rangarajan, C (2008): “Report of the Committee on

chusetts: MIT Press). linkages were as follows: for linkage 1 (13.6%) Baland, Jean-Marie, Rohini Somanathan and Lore Financial Inclusion”, RBI, Mumbai. Also at: 11.2%; for linkage 2 (71.7%) 72.6%; for linkage 3 Vandewalle (2008): “Microfinance Lifespans: A (14.7%) 16.2%. A two sample t-test of proportions Study of Attrition and Exclusion in Self-Help Rangarajan%20Commitee%20report%20on%20 confirmed no difference b etween the two. Groups in India”, India Policy Forum, 4(1): 159-210. Financial%20Inclusion.pdf 12 The following districts from these states were Bali Swain, Ranjula and Fan Yang Wallentin (2009): Srinivasan, N (2009): Microfinance India: State of the s elected – Medak and Rangareddy from Andhra “Does Microfinance Empower Women? Evidence Sector Report 2008 (New Delhi: Sage Publications). Pradesh, Dharamapuri and Villupuram from Tamil from Self Help Groups in India”, International Tankha, Ajay (2002): “Self-Help Groups as Financial Nadu, Koraput and Rayagada from Orissa, Alla-R eview of Applied Economics, 23 (5): 541-56. Intermediaries in India: Cost of Promotion, habad and Rae Bareli from Uttar Pradesh, and Centre for Global Development (2006): “When Will Sustainability, and Impact”, Interchurch Organi-Gadchiroli and Chandrapur from Maharashtra. The We Ever Learn? Report of the Evaluation Gap sation for Development Cooperation and Cordaid, states were chosen to achieve some semblance of Working Group”, mimeo, CGD, Washington DC. The Netherlands.

Economic Political Weekly

march 12, 2011 vol xlvi no 11 57

Dear Reader,

To continue reading, become a subscriber.

Explore our attractive subscription offers.

Click here


To gain instant access to this article (download).

Pay INR 200.00

(Readers in India)

Pay $ 12.00

(Readers outside India)

Back to Top