2009年10月5日月曜日

Pythonのソースコードを字句解析する

tokenizeモジュールのgenerate_tokens関数を使う

# -*- coding: utf-8 -*-

import tokenize
import StringIO
import token

src = """s = "hello"
print s"""

# generate_tokensの引数は呼び出し可能オブジェクト
g = tokenize.generate_tokens(StringIO.StringIO(src).readline)

print "#ソースコード"
print src
print
print "#トークン (type, value, start(row, col), end(row, col), line)"
for t in g:
    print t
print
print "#トークンのタイプ"
print token.tok_name

実行すると
#ソースコード
s = "hello"
print s

#トークン (type, value, start(row, col), end(row, col), line)
(1, 's', (1, 0), (1, 1), 's = "hello"\n')
(51, '=', (1, 2), (1, 3), 's = "hello"\n')
(3, '"hello"', (1, 4), (1, 11), 's = "hello"\n')
(4, '\n', (1, 11), (1, 12), 's = "hello"\n')
(1, 'print', (2, 0), (2, 5), 'print s')
(1, 's', (2, 6), (2, 7), 'print s')
(0, '', (3, 0), (3, 0), '')

#トークンのタイプ
{0: 'ENDMARKER', 256: 'NT_OFFSET', 2: 'NUMBER', 3: 'STRING', 4: 'NEWLINE', 5: 'INDENT', 6: 'DEDENT', 7: 'LPAR', 8: 'RPAR', 9: 'LSQB', 10: 'RSQB', 11: 'COLON', 12: 'COMMA', 13: 'SEMI', 14: 'PLUS', 15: 'MINUS', 16: 'STAR', 17: 'SLASH', 18: 'VBAR', 19: 'AMPER', 20: 'LESS', 21: 'GREATER', 22: 'EQUAL', 23: 'DOT', 24: 'PERCENT', 25: 'BACKQUOTE', 26: 'LBRACE', 27: 'RBRACE', 28: 'EQEQUAL', 29: 'NOTEQUAL', 30: 'LESSEQUAL', 31: 'GREATEREQUAL', 32: 'TILDE', 33: 'CIRCUMFLEX', 34: 'LEFTSHIFT', 35: 'RIGHTSHIFT', 36: 'DOUBLESTAR', 37: 'PLUSEQUAL', 38: 'MINEQUAL', 39: 'STAREQUAL', 40: 'SLASHEQUAL', 41: 'PERCENTEQUAL', 42: 'AMPEREQUAL', 43: 'VBAREQUAL', 44: 'CIRCUMFLEXEQUAL', 45: 'LEFTSHIFTEQUAL', 46: 'RIGHTSHIFTEQUAL', 47: 'DOUBLESTAREQUAL', 48: 'DOUBLESLASH', 49: 'DOUBLESLASHEQUAL', 50: 'AT', 1: 'NAME', 52: 'ERRORTOKEN', 53: 'COMMENT', 54: 'NL', 51: 'OP'}


詳細はドキュメントで

16 件のコメント:

  1. I do not even know how I ended up here, but I thought this
    post was great. I don't know who you are but definitely you're going to a famous blogger if you aren't already ;) Cheers!
    Also see my page: diets That work For women

    返信削除
  2. I'm not sure exactly why but this blog is loading extremely slow
    for me. Is anyone else having this problem or is it a issue on my end?
    I'll check back later and see if the problem still exists.



    My web blog best local roofer

    返信削除
  3. beats lamborghini
    Way cool! Some extremely valid points! I appreciate
    you writing this post and also the rest of the website is also really
    good.

    返信削除
  4. " Conclusion: The Starkist folks are real starkers. It looks like she's been heavily tranquilized and is posing while being propped up from behind in a wind tunnel. These are just a few and many more can be experienced by you when you check a game or two which you can download from secure and reliable sites and enjoy as much as you want without having to pay for it.

    Also visit my site; Mutants Genetic Gladiators Cheats

    返信削除
  5. Write more, thats all I have to say. Literally, it seems
    as though you relied on the video to make your point. You definitely know what youre talking about, why waste your intelligence on just posting videos to your
    site when you could be giving us something enlightening to read?



    Here is my site ... health benefits of garcinia cambogia

    返信削除
  6. Hiya ѵeгy cool web site!! Guy .. Excellent ..
    Wondeгful .. I will bookmark your website аnd takme the feеds
    also? I am satisfieԁ to seek out numeroսs helpful іnfo
    here within the ƿublish, we'd lіke work out more stгztegies in this
    regard, thanks for sharing. . . .. .

    返信削除
  7. Please let me know if you're looking for a author for your blog.
    You have some really great posts and I believe I would be a good asset.
    If you ever want to take some of the load off, I'd really like
    to write some articles for your blog in exchange for a link back to mine.
    Please shoot me an e-mail if interested. Thanks!

    Visit my weblog: Camp Hill child care resources

    返信削除
  8. I do not know whether it's just me or if everyone else encountering issues with your blog.
    It looks like some of the written text in your content are
    running off the screen. Can somebody else please provide feedback and let me know if
    this is happening to them too? This might be a problem with my internet browser because I've had this happen previously.
    Thanks

    返信削除
  9. Excellent blog here! Also your web site loads up
    fast! What host are you using? Can I get your affiliate link to your
    host? I wish my site loaded up as fast as yours lol

    my weblog - electrician toronto beaches

    返信削除
  10. Appreciation to my father who told me on the topic of this web site, this blog
    is truly remarkable.

    my page :: aeria points hack (http://aeriapointshackfree.blogspot.com)

    返信削除
  11. Baby of Turtle: Mother turtle lays about hundred round
    eggs. Like morphine, they reduce pain and induce feelings
    of pleasure, but without the dangerous side effects. Permission to publish this content was given by Sean Fleeman.

    返信削除
  12. It topk two үears to propcess tɦe ϲase and tthe sentence delivered ԝas 30 months, Ьut did nott ϲome սntil thе most ɑppropriate day fοr
    thiѕ family tat put so mսch trust іn tҺeir faith іn God'.
    Thе In-Cɑr 7"-9 Inch Portable DVD Player Case by USA Gear is agtain a compact, thick-padded, multi-functional case that doubles up as a carrying case for your portable DVD player and a mounting case foor placing your DVD Player on the back seat of your car. Whenn it comes tto stylish design and image quality this could well be the bst plain old DVD player thuat Sony makes. Some popular brand names are Behringer, Mackie, and Yamaha. Also, should yyou plan on driving or having a lengthy ride to destinations, you will want to bring a blanket for their child too be comfy, and perhaps a neck pillow to mzke their youngster more comfy.

    返信削除
  13. Wow, thiks piece of wriring is pleasant, mmy sister is analyzig these kinnds of things, thus I aam goiing to convey her.


    Here iss myy blog post; muscle supplements for skinny guys

    返信削除
  14. Hi it's me, I am also visiting this website on a
    regular basis, this web site is in fact good and the visitors are in fact sharing pleasant thoughts.


    My webpage: asistenta sociala

    返信削除
  15. Simply want to say your article is as astonishing.
    The clarity in your post is just spectacular and
    i can assume you're an expert on this subject.
    Well with your permission allow me to grab your RSS feed to keep
    up to date with forthcoming post. Thanks a million and please
    keep up the enjoyable work.

    Look at my blog :: Unblock Us Review

    返信削除
  16. celine bags online
    Howdy! I could have sworn I've visited your blog before but
    after browsing through some of the articles I realized it's new to me.
    Anyhow, I'm certainly pleased I found it and I'll be book-marking it and checking back frequently!

    返信削除