2.10.2011

Bibliometrics and book shrink

In the middle of doing a bunch of bibliometrics work I came across a site called BookShrink that parses a text to choose the sentences in it that are most representative of the whole. The alogorithm basically assigns a score to each sentence based on the total frequency count of all words in the book and then normalizes by sentence length. So no, it doesn't look like it has anything positional and no, it doesn't condense the book into Cliff Notes-style summary (and it doesn't read pdf's, so don't think you're getting out of doing your lit review using just some bash scripts) BUT it is pretty cool. Loading in the entirety of The Wealth of Nations yields:
1 (1.000000): The taxes upon expense, therefore, which fall chiefly upon that of the superior ranks of people, upon the smaller portion of the annual produce, are likely to be much less productive than either those which fall indifferently upon the expense of all ranks, or even those which fall chiefly upon that of the inferior ranks, than either those which fall indifferently upon the whole annual produce, or those which fall chiefly upon the larger portion of it.

2 (0.961065): I shall endeavour to give the best account I can, first, of those taxes which, it is intended should fall upon rent; secondly, of those which, it is intended should fall upon profit; thirdly, of those which, it is intended should fall upon wages; and fourthly, of those which, it is intended should fall indifferently upon all those three different sources of private revenue.

3 (0.765899): From the end of the first to the beginning of the second Carthaginian war, the armies of Carthage were continually in the field, and employed under three great generals, who succeeded one another in the command; Amilcar, his son-in-law Asdrubal, and his son Annibal: first in chastising their own rebellious slaves, afterwards in subduing the revolted nations of Africa; and lastly, in conquering the great kingdom of Spain.

4 (0.708201): You must pay, too, for the tax upon the salt, upon the soap, and upon the candles which those workmen consume while employed in your service; and for the tax upon the leather, which the saltmaker, the soap-maker, and the candle-maker consume, while employed in their service.
...
No, nothing about pins, but still pretty interesting.

No comments:

Post a Comment