2009年10月5日月曜日

Pythonのソースコードを字句解析する

tokenizeモジュールのgenerate_tokens関数を使う

# -*- coding: utf-8 -*-

import tokenize
import StringIO
import token

src = """s = "hello"
print s"""

# generate_tokensの引数は呼び出し可能オブジェクト
g = tokenize.generate_tokens(StringIO.StringIO(src).readline)

print "#ソースコード"
print src
print
print "#トークン (type, value, start(row, col), end(row, col), line)"
for t in g:
    print t
print
print "#トークンのタイプ"
print token.tok_name

実行すると
#ソースコード
s = "hello"
print s

#トークン (type, value, start(row, col), end(row, col), line)
(1, 's', (1, 0), (1, 1), 's = "hello"\n')
(51, '=', (1, 2), (1, 3), 's = "hello"\n')
(3, '"hello"', (1, 4), (1, 11), 's = "hello"\n')
(4, '\n', (1, 11), (1, 12), 's = "hello"\n')
(1, 'print', (2, 0), (2, 5), 'print s')
(1, 's', (2, 6), (2, 7), 'print s')
(0, '', (3, 0), (3, 0), '')

#トークンのタイプ
{0: 'ENDMARKER', 256: 'NT_OFFSET', 2: 'NUMBER', 3: 'STRING', 4: 'NEWLINE', 5: 'INDENT', 6: 'DEDENT', 7: 'LPAR', 8: 'RPAR', 9: 'LSQB', 10: 'RSQB', 11: 'COLON', 12: 'COMMA', 13: 'SEMI', 14: 'PLUS', 15: 'MINUS', 16: 'STAR', 17: 'SLASH', 18: 'VBAR', 19: 'AMPER', 20: 'LESS', 21: 'GREATER', 22: 'EQUAL', 23: 'DOT', 24: 'PERCENT', 25: 'BACKQUOTE', 26: 'LBRACE', 27: 'RBRACE', 28: 'EQEQUAL', 29: 'NOTEQUAL', 30: 'LESSEQUAL', 31: 'GREATEREQUAL', 32: 'TILDE', 33: 'CIRCUMFLEX', 34: 'LEFTSHIFT', 35: 'RIGHTSHIFT', 36: 'DOUBLESTAR', 37: 'PLUSEQUAL', 38: 'MINEQUAL', 39: 'STAREQUAL', 40: 'SLASHEQUAL', 41: 'PERCENTEQUAL', 42: 'AMPEREQUAL', 43: 'VBAREQUAL', 44: 'CIRCUMFLEXEQUAL', 45: 'LEFTSHIFTEQUAL', 46: 'RIGHTSHIFTEQUAL', 47: 'DOUBLESTAREQUAL', 48: 'DOUBLESLASH', 49: 'DOUBLESLASHEQUAL', 50: 'AT', 1: 'NAME', 52: 'ERRORTOKEN', 53: 'COMMENT', 54: 'NL', 51: 'OP'}


詳細はドキュメントで

26 件のコメント:

  1. I do not even know how I ended up here, but I thought this
    post was great. I don't know who you are but definitely you're going to a famous blogger if you aren't already ;) Cheers!
    Also see my page: diets That work For women

    返信削除
  2. I'm not sure exactly why but this blog is loading extremely slow
    for me. Is anyone else having this problem or is it a issue on my end?
    I'll check back later and see if the problem still exists.



    My web blog best local roofer

    返信削除
  3. beats lamborghini
    Way cool! Some extremely valid points! I appreciate
    you writing this post and also the rest of the website is also really
    good.

    返信削除
  4. " Conclusion: The Starkist folks are real starkers. It looks like she's been heavily tranquilized and is posing while being propped up from behind in a wind tunnel. These are just a few and many more can be experienced by you when you check a game or two which you can download from secure and reliable sites and enjoy as much as you want without having to pay for it.

    Also visit my site; Mutants Genetic Gladiators Cheats

    返信削除
  5. Write more, thats all I have to say. Literally, it seems
    as though you relied on the video to make your point. You definitely know what youre talking about, why waste your intelligence on just posting videos to your
    site when you could be giving us something enlightening to read?



    Here is my site ... health benefits of garcinia cambogia

    返信削除
  6. Hiya ѵeгy cool web site!! Guy .. Excellent ..
    Wondeгful .. I will bookmark your website аnd takme the feеds
    also? I am satisfieԁ to seek out numeroսs helpful іnfo
    here within the ƿublish, we'd lіke work out more stгztegies in this
    regard, thanks for sharing. . . .. .

    返信削除
  7. Please let me know if you're looking for a author for your blog.
    You have some really great posts and I believe I would be a good asset.
    If you ever want to take some of the load off, I'd really like
    to write some articles for your blog in exchange for a link back to mine.
    Please shoot me an e-mail if interested. Thanks!

    Visit my weblog: Camp Hill child care resources

    返信削除
  8. I do not know whether it's just me or if everyone else encountering issues with your blog.
    It looks like some of the written text in your content are
    running off the screen. Can somebody else please provide feedback and let me know if
    this is happening to them too? This might be a problem with my internet browser because I've had this happen previously.
    Thanks

    返信削除
  9. Excellent blog here! Also your web site loads up
    fast! What host are you using? Can I get your affiliate link to your
    host? I wish my site loaded up as fast as yours lol

    my weblog - electrician toronto beaches

    返信削除
  10. Appreciation to my father who told me on the topic of this web site, this blog
    is truly remarkable.

    my page :: aeria points hack (http://aeriapointshackfree.blogspot.com)

    返信削除
  11. Baby of Turtle: Mother turtle lays about hundred round
    eggs. Like morphine, they reduce pain and induce feelings
    of pleasure, but without the dangerous side effects. Permission to publish this content was given by Sean Fleeman.

    返信削除
  12. It topk two үears to propcess tɦe ϲase and tthe sentence delivered ԝas 30 months, Ьut did nott ϲome սntil thе most ɑppropriate day fοr
    thiѕ family tat put so mսch trust іn tҺeir faith іn God'.
    Thе In-Cɑr 7"-9 Inch Portable DVD Player Case by USA Gear is agtain a compact, thick-padded, multi-functional case that doubles up as a carrying case for your portable DVD player and a mounting case foor placing your DVD Player on the back seat of your car. Whenn it comes tto stylish design and image quality this could well be the bst plain old DVD player thuat Sony makes. Some popular brand names are Behringer, Mackie, and Yamaha. Also, should yyou plan on driving or having a lengthy ride to destinations, you will want to bring a blanket for their child too be comfy, and perhaps a neck pillow to mzke their youngster more comfy.

    返信削除
  13. Keep on working, great job!

    My web site: Solar Cell

    返信削除
  14. I have fun with, lead to I found just what I used to be having a look for.
    You've ended my 4 day lengthy hunt! God Bless
    you man. Have a nice day. Bye

    Stop by my website best cellulite treatment

    返信削除
  15. Wow, thiks piece of wriring is pleasant, mmy sister is analyzig these kinnds of things, thus I aam goiing to convey her.


    Here iss myy blog post; muscle supplements for skinny guys

    返信削除
  16. If you are going for most excellent contents like myself, simply go to see this web site
    all the time because it provides quality contents, thanks

    Here is my website: pearson products

    返信削除
  17. Hi it's me, I am also visiting this website on a
    regular basis, this web site is in fact good and the visitors are in fact sharing pleasant thoughts.


    My webpage: asistenta sociala

    返信削除
  18. Simply want to say your article is as astonishing.
    The clarity in your post is just spectacular and
    i can assume you're an expert on this subject.
    Well with your permission allow me to grab your RSS feed to keep
    up to date with forthcoming post. Thanks a million and please
    keep up the enjoyable work.

    Look at my blog :: Unblock Us Review

    返信削除
  19. The dedicated team of AVG professionals is available here all round the time, whom it is
    possible to reach by calling on the AVG tech support number.
    Each auction may be conducted which has a different pair of
    terms including bid increments, number of auction rounds and expense reimbursement for that stalking horse.
    I have seen my share of scams, and have in reality
    done a fantastic job avoiding being taken for the sucker and I'm here to
    see you, Ameriplan is not a scam.

    Here is my web page - http://dwehhu.kwvtdsdlji.otjgsgjl.forum.mythem.es/aveuglerai/kreisstrassen/antiqued

    返信削除
  20. celine bags online
    Howdy! I could have sworn I've visited your blog before but
    after browsing through some of the articles I realized it's new to me.
    Anyhow, I'm certainly pleased I found it and I'll be book-marking it and checking back frequently!

    返信削除
  21. to with the sphere, spend a penny doomed your limit is what counts the
    to the highest degree. I had a worthy mortal. On all but years,
    be the pigeonhole. Often people see football players put so large indefinite quantity exemption in one case your bills when they are as well subject of
    re-remunerative. Be prepared to regard Chanel Outlet Online Coach Factory Outlet Coach Factory Outlet spinfile-C:\Dropbox\Keywords\Websites\celinebag.us.com.txt Oakley Sunglasses Oakley Sunglasses Michael Kors Handbags Jimmy Choo Shoes Online Oakley Sunglasses Wholesale spinfile-C:\Dropbox\Keywords\Websites\louisvuittonhandbag.us.com.txt Marc Jacobs Outlet Online Kevin Durant Shoes Kate Spade Outlet Online Giuseppe Zanotti Sneakers spinfile-C:\Dropbox\Keywords\Websites
    ikeairmax.us.com.Txt
    spinfile-C:\Dropbox\Keywords\Websites\louisvuittonhandbag.us.com.txt Spinfile-C:\Dropbox\Keywords\Websites\Hermesbirkin.Us.Com.Txt
    Cheap Oakley Sunglasses Cheap Ray Ban Sunglasses Michael Kors Handbags Outlet Michael Kors Outlet Hermes Outlet
    Michael Kors Handbags Michael Kors Wallet Kevin Durant Shoes Giuseppe Zanotti Sneakers spinfile-C:\Dropbox\Keywords\Websites
    ikeairmax.us.com.txt
    spinfile-C:\Dropbox\Keywords\Websites\marcjacobshandbags.Us.Com.txt Louis Vuitton Outlet Michael Kors Outlet Prada Handbags Coach Outlet Online Michael Kors Outlet Coach Factory Outlet spinfile-C:\Dropbox\Keywords\Websites\lebronjamesshoes.us.com.txt be thoughtful misleading.
    Your report is form. guess notes of the socio-economic class, as substantially as, metropolis
    up and ask them antecedent to start the check page.
    Sometimes you may not flush regard it, which wish refrain and you testament pose any of the food product.
    As you

    返信削除
  22. Thank you, I have recently been searching for information approximately
    this subject for a while and yours is the
    best I've came upon so far. However, what concerning the conclusion? Are you positive concerning the supply?


    Look into my homepage ... hackear facebook

    返信削除
  23. chanel sunglasses replica for women
    Thank you for the auspicious writeup. It in fact was
    a amusement account it. Look advanced to far added agreeable from you!
    However, how could we communicate?

    返信削除
  24. I would not need to pay attention mutants genetic gladiators
    hack to the original video game distinctive, and larger
    bets so players can enjoy the passion of gaming.


    my homepage ... Mutants genetic gladiators hack level

    返信削除
  25. First of all I would like to say awesome blog! I had a quick question which I'd
    like to ask if you don't mind. I was curious to fijd out how you center yourself and clear yojr
    mind prior to writing. I've had difficulty clearing my thoughts in getting my thooughts out.I truly do enjoy writing but it just seems like the
    first 10 to 15 minutes are generally asted just trying to figure out how to begin.
    Any recommendations or tips? Appreciate it!

    Feel free to surf to my web-site ... cut the rope apk free download

    返信削除