From: Dan Sugalski At 11:03 AM 1/17/01 -0200, Branden wrote:
>Hi.
>
>I'm actually not following this list from close and I searching the archives
>isn't that easy yet, so pardon me if this was already brought up.

As Simon's pointed out, there's an RFC for this already.

It's actually not that tough to do now with eval, with two problems:

*) Eval won't extend scope, so each statement you execute stands alone. (So
lexicals go missing immediately)
*) Figuring out where a statement ends is a pain--you'd want to read to the
end of a block for example before executing it.

I'd personally like to provide a scopeless string eval, and the second
issue's a SMOP. (Though involving the parser on the fly would make it easier)

Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
dan@sidhe.org have teddy bears and even
teddy bears get drunk

From: Tim Jenness On Wed, 17 Jan 2001, Branden wrote:

> I'm actually not following this list from close and I searching the archives
> isn't that easy yet, so pardon me if this was already brought up.
>
> I work with Perl and I also work with Tcl, and one thing I actually like
> about Tcl is that it's interactive like a shell, i.e. it gives you a prompt,
> where you type commands in and, if you type a whole command by the end of
> the line, it executes it, otherwise, it gives you another prompt and keeps
> reading more lines until the whole command is typed, when it's executed. I
> think this is particularly useful for:
> a) testing features (what the value of ... would be if I ...?)

This may not sound obvious but I use the Perl Data Language (PDL) for
interactive testing since it provides a shell interface. It's not perfect
(needs to be extended to handle multi-line input) but it's a start.

Both Python and Tcl support features like this and I agree that a perl
shell would be useful.

--
Tim Jenness
JCMT software engineer/Support scientist
http://www.jach.hawaii.edu/~timj

From: Simon Cozens On Wed, Jan 17, 2001 at 11:03:05AM -0200, Branden wrote:
> I work with Perl and I also work with Tcl, and one thing I actually like
> about Tcl is that it's interactive like a shell, i.e. it gives you a prompt,
> where you type commands in and, if you type a whole command by the end of
> the line, it executes it, otherwise, it gives you another prompt and keeps
> reading more lines until the whole command is typed, when it's executed.

Sounds like http://dev.perl.org/rfc/184.html to me.

--
Do not go gentle into that good night/Rage, rage against the dying of the light

From: Branden Hi.

I'm actually not following this list from close and I searching the archives
isn't that easy yet, so pardon me if this was already brought up.

I work with Perl and I also work with Tcl, and one thing I actually like
about Tcl is that it's interactive like a shell, i.e. it gives you a prompt,
where you type commands in and, if you type a whole command by the end of
the line, it executes it, otherwise, it gives you another prompt and keeps
reading more lines until the whole command is typed, when it's executed. I
think this is particularly useful for:
a) testing features (what the value of ... would be if I ...?)
b) debugging (what is the value of the variable ...?/what would the function
... return if I give it value ...?) and building custom debuggers
c) learning modules' interfaces
d) making changes at runtime (I will change the value of this variable
now...)

I understand the Perl 5's debugger can handle some of these tasks, but I
argue that its support is quite incomplete, it's hard to do if's or while
loops, and it's very hard (I would say impossible) to customize the debugger
for some of the uses above.

Tcl manages this by having a function that gets a line of command and
returns wheter it's complete, or another line should be read to try to
complete it. Of course Tcl can do it because it's much simpler than Perl in
its syntax, having this done in Perl is a much more complex subject.

Still, I think it's worth it. And now is for sure the time to think on this
kind of thing. After the implementation is done, if it's there, it's there,
otherwise, it will probably never be!

Thanks a lot.

Branden

From: Simon Cozens On Tue, Jan 02, 2001 at 06:22:31PM -0800, Steve Fink wrote:
> Just in looking at your example, it seems like some complex replacements
> would be a bit of a pain to generate. It would be nice to have a
> specification for those opcode replacements. Like, say, perl code. How
> hard would it be to do:

Trivial. Use B::svref_2object(sub { your code here }) to create an op tree in
a variable. Splice it in as before.

--
"There is no statute of limitations on stupidity."
-- Randomly produced by a computer program called Markov3.

From: Steve Fink That is way cool.

Though I'm not sure that all of the constructs of another language are
going to be that easy to map into perl opcodes. Arithmetic, sure. But
perl opcodes aren't exactly a universal intermediate language.

Just in looking at your example, it seems like some complex replacements
would be a bit of a pain to generate. It would be nice to have a
specification for those opcode replacements. Like, say, perl code. How
hard would it be to do:

use constant SUBTRACT_TWO_THINGS => B::eval(<<'END');
print("Subtracting!!!\n"),
$XYZZY_PLACEHOLDER_PARAM1-$XYZZY_PLACEHOLDER_PARAM2
END

CHECK {
.
.
.
$z = new B::subst_op(SUBTRACT_TWO_THINGS, XYZZY_PLACEHOLDER_PARAM1 =>
$x->first, XYZZY_PLACEHOLDER_PARAM2 => $x->last);
.
.
.
}

From: Simon Cozens On Mon, Dec 18, 2000 at 10:12:35AM -0500, Andy Dougherty wrote:
> The issues of 'use Python' or 'use Pythonish' are a quite different issue.
> I don't think anyone believes it ought to be easy to *write* the Pythonish
> module. But it ought to be *possible*.

Incidentally, it's possible in Perl 5. Very possible; I'm just about to
finish off a module called B::Generate which allows you to create your
own op trees - it's basically B but with get/set methods instead of get
methods, plus constructors for all the classes.

Although I haven't tested this yet, if you have a source filter which
implements a parser for your favourite language and uses B::Generate to spit
out an op tree and point Perl at the root of it, instant Python interpreter!

Well, maybe not exactly *instant*, but... :)

As a very simple example, the following program prints "-5" by replacing
"add" with "subtract":

#!/usr/bin/perl
use B::Generate;

CHECK{
my ($x, $y,$z);
$x = B::main_start;
for ($x = B::main_start; $x->type != 62; $x=$x->next){ # Find "add"
$y=$x; # $y is the op before "add"
};
$z = new B::BINOP("subtract",0,$x->first, $x->last); # Create replacement
$z->next($x->next); # Copy add's "next" across.
$y->next($z); # Tell $y to point to replacement op.
}

$a = 10; $b = 15; print $a + $b;


--
Only two things are infinite: the Universe and human stupidity, and I'm
not sure about the former - Albert Einstein

From: Steve Fink I'm wondering if we should explicitly break out the languages that
comprise perl today. That'd be at least toplevel perl, regular
expressions, and pack. Maybe tr// and the second half of s/// are
sufficiently different too. If nothing else, it would highlight the
problems in switching languages midstream. Like:

*) can you always know that you're definitely switching languages, or
might you have to back out of a regular expression parse when you
realize you've been reading a comment all along? Related: will you
sometimes need a lookahead "token" or twelve to figure out what's about
to happen from the outer language's perspective?

*) When you suspend the parse, you may need to suspend a whole tree of
language parsers.

*) Tokenization can be totally different for different languages.

*) Tokenization and parsing can depend on the interpreter's state. (Good
example? Indirect objects?) We either need an API for accessing such
state in whatever wacky structures the sub-languages store it in, or
define some rules about what cannot affect a parse. Gets into language
definition.

*) The end of a mini-language may have to be determined by agreement
between the outer and inner languages (the inner language has to agree
that it's at a stopping point; the outer language has to agree that the
next stuff can end an inner language region.)

*) These languages recurse quite a bit. Think s///e, or just variable
interpolation: m!$x{foo(3**1e6, "An entire program")}!. Though that
example will fail in perl5 if you say "An entire program!"; should it?
(If the answer is yes, then the outer language either has to tell the
inner "I know that normally you're okay with an unescaped exclamation
point, but don't be, please?", or it must impose its own notions of
escaping on the inner language to be able to scan ahead for the ending.
If the answer is no, then highlighting editors will be incrementally
more unhappy.)

From: Ask Bjoern Hansen On Sun, 17 Dec 2000, Simon Cozens wrote:

> This is the fourth time I've sent this mail to perl6-internals-api-parser,

I've seen it on the list at least twice.

> but it doesn't seem to be arriving. None of my other mail is affected, and
> perl5-porters is, for once, behaving itself; why this list in particular?

Send me the log snippets from your mailserver successfully
delivering the mail to one of the perl.org mx'es and I can look
after it.

If it was before the weekend, don't bother though. The 70MB's I have
for qmail logs rotate pretty quickly.

So maybe this is better: Next time you think it happens, send me a
mail right away and I can look into it.

(perl.org haven't ever dropped mail since I started taking care of
it as far as I know. Are you sure that you'll receive bounces, your
envelope sender is often changing).


- ask

--
ask bjoern hansen - <http://ask.netcetera.dk/>
more than 70M impressions per day, <http://valueclick.com>

From: David Grove
Andy Dougherty <doughera@lafayette.edu> wrote:

> On Mon, 18 Dec 2000, David Grove wrote:
>
> >
> > Andy Dougherty <doughera@lafayette.edu> wrote:
> >
> I think you misunderstand. I think it should be very easy to *use* a
> hypothetical Pythonish module. I don't expect it will be very easy to
> create it, and I don't see it as worthwhile to expend a
disproportionate
> amount of effort in that direction.
>
> In another message, you write ...
>
> > I also admit that I would, on a purely personal-bias level, prefer
not
> > to
> > cast too much support in the direction of Python, Java, C#, or ASP at
>
> which also seems to aruge for not working too too hard to make it easy
to
> write the Pythonish module.

As for how easy it is to write one, that's probably just going to come out
in development. My hope is that we can make it as straightforward as
possible for the user. I do hope to help guide it in the direction of user
laziness and hubris.

The bias expressed is one like many hold but don't express. The two
statements don't really relate, and were said in two different stages of
shock. The bias is, I suppose, one of fear. At this point I'm not sure
whether it would help or hurt to "support" the languages in their full
spec rather than just use some of their features to enhance Perl. I'll
rely on my "elders'" opinions in this area. I've seen this debate fought
in other areas, and there are valid concerns in both directions (to
swollow it up, or not to swollow it up). This isn't the place for that, so
please leave that as an explanation of my bias and not as a topic
direction.

> I think one or both of us is confused.

I was. I conceded the issue, however. It seems that what was in my head
was very close to what we'll likely come up with, but I just hadn't
envisioned it going so far. I also think it will be, or can be, much
easier for both us and the user than I first thought.

> > I don't
> > see these little languages as Perl features, but as programmer
features,
>
> I don't see Python or Java as a "little language". Perhaps that's the
> source of the confusion. I see a whole spectrum of "languages" one
might
> want to feed to perl6. Easy ones ("little languages") should be easy.
> Hard ones (e.g. Tcl, Python) should be possible. In-between-ones
should
> be in between.

Correct. We're riding the same frequency wave at this point, I think.

Thanks for your help in correcting my misconceptions of our directions.

p

From: Andy Dougherty On Mon, 18 Dec 2000, David Grove wrote:

>
> Andy Dougherty <doughera@lafayette.edu> wrote:
>
> > The issues of 'use Python' or 'use Pythonish' are a quite different
> issue.
> > I don't think anyone believes it ought to be easy to *write* the
> Pythonish
> > module.
>
> I do.

> That's the problem. This is a nearly ubiquitously desired objective
> (writing the modules or whatever they are), but I have the fear that
> actually writing one will be so undaunting that it will be a seldom-used
> feature, or one that will be as often avoided as complex perl OOP (beyond
> the basics), provided only by the highest levels of perl mastery.

I think you misunderstand. I think it should be very easy to *use* a
hypothetical Pythonish module. I don't expect it will be very easy to
create it, and I don't see it as worthwhile to expend a disproportionate
amount of effort in that direction.

In another message, you write ...

> I also admit that I would, on a purely personal-bias level, prefer not
> to
> cast too much support in the direction of Python, Java, C#, or ASP at

which also seems to aruge for not working too too hard to make it easy to
write the Pythonish module.

I think one or both of us is confused.

> I don't
> see these little languages as Perl features, but as programmer features,

I don't see Python or Java as a "little language". Perhaps that's the
source of the confusion. I see a whole spectrum of "languages" one might
want to feed to perl6. Easy ones ("little languages") should be easy.
Hard ones (e.g. Tcl, Python) should be possible. In-between-ones should
be in between.

--
Andy Dougherty doughera@lafayette.edu
Dept. of Physics
Lafayette College, Easton PA 18042

From: Nick Ing-Simmons Sam Tregar <sam@tregar.com> writes:
>
>It comes down to what is meant by "little language". When I hear that
>term I immediately think Scheme and TCL. They both have a small core and
>extremely regular syntax. I can imagine writing a smallish parser that
>spits out Perl bytecode for either.

TCL is surprisingly hard to get right. The parse is trivial
but typical Tcl code uses the quoting mechanisms to creatively
create variable names at the various interpolation times.

This kind of mess (syntax from rusty memory):

proc messy {name suffix level value} {
set command "$name$suffix"
upvar $command [expr $level-2]
set command "$command {$value}"
set $command
}

messy foo "[$index]" 3 {expr $command+1}

That "obviously" compiles to the bytecode for

$foo{$index}++

;-)

--
Nick Ing-Simmons <nik@tiuk.ti.com>
Via, but not speaking for: Texas Instruments Ltd.

From: Nick Ing-Simmons Bradley M . Kuhn <bkuhn@ebb.org> writes:
>Nicholas Clark <nick@ccl4.org> wrote:
>
>> Something I though of:
>> If you're trying to write an interactive perl inputer - either a perl shell
>> or just the command prompt on the debugger it would be useful if you
>> could tell the parser that the chunk of source you're giving it may be
>> incomplete.
>
>I really like this idea, although I am unsure of how we might implement it.
>I don't recall any of the compiler texts I have read over the years talking
>about formal methods for writing such a "partial programs are acceptable"
>parser.
>
>
>Lisp-like languages handle this, but s-expressions are so trivial to parse
>that it's no help to simply "follow" their example.

So does Tcl - but it is also trivial to "parse" - all its complexity
is in the semantics of interpolation.

--
Nick Ing-Simmons <nik@tiuk.ti.com>
Via, but not speaking for: Texas Instruments Ltd.

From: Nick Ing-Simmons David Grove <pete@petes-place.com> writes:
>
>Because what is the parser/lexer/tokenizer parsing? Perl? Pythonic?
>Javanese?

It is entierly possible to use one parser/lexer "engine" for multiple
languages - for example a yacc/byacc/bison LALR(1) parser is a simple
state machine - all the language stuff is in the "data" part.
But switching lexer and parser's look-ahead "token" at language
boundaries is a tad tricky. But that is probably not what is being discussed
here.

(Nick just passing through...)

--
Nick Ing-Simmons <nik@tiuk.ti.com>
Via, but not speaking for: Texas Instruments Ltd.

From: David Grove
Dan Sugalski <dan@sidhe.org> wrote:

> At 12:29 PM 12/18/00 +0000, David Grove wrote:
>
> >Sam Tregar <sam@tregar.com> wrote:
> >
> > > On Mon, 18 Dec 2000, David Grove wrote:
> > >
> >
> >[snip]
> >
> > > > _Perl_ _within_ _a_ _Perl_ _context_ _and_ _for_ _Perl_
_purposes_,
> > >
> > > Feeling a little hostile to the rest of the programming world?
You're
> > > sounding almost nationalistic! We're not at war.
> >
> >No no. I just think we've redefined the "little languages" as everests
> >rather than molehills. That scares me.
>
> Luckily, Dave, this is a matter of perception on your part. :) Not that
> the
> concern's not worth having, just that I don't think it's going to be an
> issue.

I'm starting to get that impression. I feel like:

1. Ok, the car's broken, we're going to have to get it to the service
station. There's one right over there.

2. I find out we have to push it. YIKES. You want to go to the _shell_
station instead??? I can see the shell station sign at the top of the
5000ft 20deg incline hill.

3. I find out we're going to the one at the bottom of the hill instead.
"What? You think we're nuts?"

I think I've gotten stuck is shock somewhere among #2 and #3 there. It's a
total change in what it seemed we were aiming for. If you guys think it's
not overbloated, I've no problem with it. However, I'm still hoping to aim
for ease of user implementation of a "little language". The user community
has always been most of my focus in Perl affairs. I don't care if it's
hard for us, as long as they can be lazy and [whatever the adjectival form
of "hubris" is].

Basically, I can foresee much of the user end of creating a "little
language" the creation of a stored hash of translations (lexeme
translations, not source translations), and probably another one to
identify those lexemes. Is this about the size of it?

BTW, as for the output, I realize the parser doesn't output java byte
code. I was referring to perl as a whole, of course.

p

From: Dan Sugalski At 06:05 PM 12/18/00 +0000, Nicholas Clark wrote:
>On Mon, Dec 18, 2000 at 11:30:09AM +0000, David Grove wrote:
> >
> > Bart Lateur <bart.lateur@skynet.be> wrote:
>
> > > But, the gist of this post is: we don't want to loose the usefulness of
> > > the syntax highlighter, as soon as there is one syntax error in the
> > > script, because this will be the normal situation while editing source.
> > > Parsers are generally very bad at parsing erroneous code.
> >
> > You're forgetting something. Any such editor would have to be written
> > either in Perl, or in C with builtin Perl, in order to gain access to this
> > type of parser feedback. That or we'd have to communicate with perl
>
>When I made the suggestion to give the perl parser enough API to let
>an editor use it for syntax highlighting, I was thinking of an editor
>written in C with embedded perl. Such as emacs or vi.

Yup. The parser should have some mechanism to parse pieces of code, so the
editor can pass in small chunks for analysis.

Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
dan@sidhe.org have teddy bears and even
teddy bears get drunk

From: Dan Sugalski At 12:29 PM 12/18/00 +0000, David Grove wrote:

>Sam Tregar <sam@tregar.com> wrote:
>
> > On Mon, 18 Dec 2000, David Grove wrote:
> >
>
>[snip]
>
> > > _Perl_ _within_ _a_ _Perl_ _context_ _and_ _for_ _Perl_ _purposes_,
> >
> > Feeling a little hostile to the rest of the programming world? You're
> > sounding almost nationalistic! We're not at war.
>
>No no. I just think we've redefined the "little languages" as everests
>rather than molehills. That scares me.

Luckily, Dave, this is a matter of perception on your part. :) Not that the
concern's not worth having, just that I don't think it's going to be an issue.

>I also admit that I would, on a purely personal-bias level, prefer not to
>cast too much support in the direction of Python, Java, C#, or ASP at the
>moment. A bit of stolen grammar, perhaps, but not support for the
>languages outside of a pure perl context. We have a bit of ground to
>recover. I've mixed feelings whether this effort can help or harm that. It
>depends on marketing, of course. That's probably something that I just
>have to work through, since I know we want that.

I don't really care if we parse any other language. (Though the hooks
should be in place to do so if someone wants to put the work forth to do
it) Emitting Java bytecode and .NET stuff's one of the design goals, but
that's for a different place.

It does mean we want to have some method of specifying access to external
libraries--Java without AWT (or whatever they're using at the moment)
wouldn't be all that much fun.

>But that's in context,
>because that's what it sounds like we're reaching for: the ability to
>parse anything to perl, whether it actually translates or not, using
>methods that seem out of scope. We also want to produce the output of run
>| binary | bytecode, meaning python in, .class (java) out.

Right, but the output is separate from the input. There will be modules to
take perl bytecode and emit .NET and java bytecode. Where the perl bytecode
came from is pretty much irrelevant.

>I think this _should_ be a simple thing, or as recently suggested a simple
>set of things. We don't appear to be going in that direction.

It can't be that simple. Not the nature of the beast--there is some
complexity involved.

>I have long since seen the "little languages" as altering Perl's syntax.

Not a view shared by other people. Which is good, because it doesn't, really.

>Parsing C
>into Perl or Perl source trees doesn't compute into that and it's just
>blowing my mind.
>
>Maybe it is simple. If it is, no problem.

I think at the moment you just have to take it on faith that it'll work.
(And it will) Perhaps once things come into clearer focus they'll gel for you.

Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
dan@sidhe.org have teddy bears and even
teddy bears get drunk

From: David Grove
Sam Tregar <sam@tregar.com> wrote:

> On Mon, 18 Dec 2000, David Grove wrote:
>

[snip]

> > _Perl_ _within_ _a_ _Perl_ _context_ _and_ _for_ _Perl_ _purposes_,
>
> Feeling a little hostile to the rest of the programming world? You're
> sounding almost nationalistic! We're not at war.

No no. I just think we've redefined the "little languages" as everests
rather than molehills. That scares me.

I also admit that I would, on a purely personal-bias level, prefer not to
cast too much support in the direction of Python, Java, C#, or ASP at the
moment. A bit of stolen grammar, perhaps, but not support for the
languages outside of a pure perl context. We have a bit of ground to
recover. I've mixed feelings whether this effort can help or harm that. It
depends on marketing, of course. That's probably something that I just
have to work through, since I know we want that.

I think that it would be a nice thing to have, too, if we could use one
interpreter for multiple languages. One library in another language's
context. But like you say, that's out of context. But that's in context,
because that's what it sounds like we're reaching for: the ability to
parse anything to perl, whether it actually translates or not, using
methods that seem out of scope. We also want to produce the output of run
| binary | bytecode, meaning python in, .class (java) out.

I think this _should_ be a simple thing, or as recently suggested a simple
set of things. We don't appear to be going in that direction. I have long
since seen the "little languages" as altering Perl's syntax. Parsing C
into Perl or Perl source trees doesn't compute into that and it's just
blowing my mind.

Maybe it is simple. If it is, no problem. I do hope that we are able to
provide a simple interface to the end user for being able to use the
feature, however. Whatever effort we have to do to provide _them_ with
this simplicity is merited. It's the complexity of their work I'm
concerned about, not mine/ours.

Ideally, we're doing nothing more than mapping one to 'nother lexeme set.
If the user can do that simply, I'm happy (and would like to branch into
actually getting that started). Otherwise, if the user can't easily use
this feature, we haven't planned well.

> > I have to be cynical about this. Of what benefit is this to the Perl
> > community and language?
>
> Yeesh. You'd think we were proposing a serious piece of work here! Of
> what benefit to the Perl community is your staunch oposition to
defining
> an API that most of us want?

That's out of context for my concerns.

Let me be a little clearer and calmer: I am _already_ working on something
that could provide and receive mutual benefit from this tiny tidbit, so,
to me, it's serious. It's also serious because it's somewhere I _can_
participate effectively. I'm concerned because we've gone from altering
how perl sees Perl, to how perl interprets the entirety of the C language.
I'd like very much to participate in working on this specific tiny tidbit,
but we seem to be at two extremes (meeting in the middle in a few posts)
of complexity and goal. In order to understand how I can participate, I'm
looking for sensible methods of achieving the goals, and sensible goals to
achieve. Parsing C-laced-perl seems sensible. Parsing C does not. I mean,
if one wanted to go that far, more power to him. Do we have the ability to
do the simple things before we try to do the unseemly? I mean, can we aim
for the first and arrive at the latter?

As I've expressed, this is an area that I have intense interest, because
of my own work. In order for me to do my own work, and by extension to
assist in this piece of the puzzle in Perl 6, I have to find clear and
sensible goals and methods of reaching those goals. So, yes, it's
important to me.

On the other hand, you have made an interesting point... I didn't think of
it before. Using a c/python/java library within Perl code to come up with
cross-language-ability... I like that.

> Is it just that you lack the imagination to see the potential benefit
> here?

I do see the benefits, but it's almost utopian. I feel like we're trying
to eat an entire cow at once raw rather than a bite of juicy filet mignon
smotherd in mushrooms.

p

From: David Grove
Nicholas Clark <nick@ccl4.org> wrote:

> On Mon, Dec 18, 2000 at 11:30:09AM +0000, David Grove wrote:
> >
> > Bart Lateur <bart.lateur@skynet.be> wrote:
>
> > > But, the gist of this post is: we don't want to loose the
usefulness
> of
> > > the syntax highlighter, as soon as there is one syntax error in
the
> > > script, because this will be the normal situation while editing
> source.
> > > Parsers are generally very bad at parsing erroneous code.
> >
> > You're forgetting something. Any such editor would have to be written
> > either in Perl, or in C with builtin Perl, in order to gain access to
> this
> > type of parser feedback. That or we'd have to communicate with perl
>
> When I made the suggestion to give the perl parser enough API to let
> an editor use it for syntax highlighting, I was thinking of an editor
> written in C with embedded perl. Such as emacs or vi.
>
> I didn't say that anyone should actually do it :-)

Too late. ;-)

> [or that it wouldn't be usably fast on anything expensive enough to
come
> with less than 1Gb RAM as standard]
> Just that it would be nice if the parser API were flexible enough to
make
> it possible (if not easy) for someone to do it.
> As it seemed to be a bit of parser API we'd not yet considered.
> And this is the parser-api list, so it seemed very on topic.
>
> I think I'm not wrong in saying that making the parser state totally
> encapsulated makes the parser restartable and goes a long way to making
> it re-entrant.

Actually, I'm fascinated with the possibilities if it's feasible. Any
objections to a private convo to come up with something feasible? I've hit
these issues pretty hard in the past. Maybe we can come up with something
completely feasible.

p

From: Nicholas Clark On Mon, Dec 18, 2000 at 11:30:09AM +0000, David Grove wrote:
>
> Bart Lateur <bart.lateur@skynet.be> wrote:

> > But, the gist of this post is: we don't want to loose the usefulness of
> > the syntax highlighter, as soon as there is one syntax error in the
> > script, because this will be the normal situation while editing source.
> > Parsers are generally very bad at parsing erroneous code.
>
> You're forgetting something. Any such editor would have to be written
> either in Perl, or in C with builtin Perl, in order to gain access to this
> type of parser feedback. That or we'd have to communicate with perl

When I made the suggestion to give the perl parser enough API to let
an editor use it for syntax highlighting, I was thinking of an editor
written in C with embedded perl. Such as emacs or vi.

I didn't say that anyone should actually do it :-)
[or that it wouldn't be usably fast on anything expensive enough to come
with less than 1Gb RAM as standard]
Just that it would be nice if the parser API were flexible enough to make
it possible (if not easy) for someone to do it.
As it seemed to be a bit of parser API we'd not yet considered.
And this is the parser-api list, so it seemed very on topic.

I think I'm not wrong in saying that making the parser state totally
encapsulated makes the parser restartable and goes a long way to making
it re-entrant.

Nicholas Clark

From: Sam Tregar On Mon, 18 Dec 2000, David Grove wrote:

> For the full language spec, I don't think it's attainable, and honestly
> don't see the reason for it within the context of Perl.

I've got a simple reason for it - I think it's going to be part of the
Perl6 spec. Do I have any proof? Nope. We'll know when Larry's finished
with the Perl6 spec.

What about this work do you see an unatainable? We're really just talking
about defining an API for people to hook into the Perl 6 parser at various
stages. If you're correct and this really is a bad idea then we will have
lost little - no one will write any add-on parsers and we'll be left with
a well-documented interface.

> to _replace_ python, gnu c, java sdk, lisp, etc. (That's what I'm
> seeing here.)

Take off your visionary goggles. Nobody is suggesting that Perl will
become the prefered platform for any of those languages. However, if I
can take a well-established Python library and drop it into my new Perl 6
project I'd be pretty pleased. If I can write the recursive-searching
algorithm for my next project in Lisp instead of Perl, who's to say I
shouldn't be able to?

Think of it as the ultimate SWIG - no .i files needed!

> _Perl_ _within_ _a_ _Perl_ _context_ _and_ _for_ _Perl_ _purposes_,

Feeling a little hostile to the rest of the programming world? You're
sounding almost nationalistic! We're not at war.

> I have to be cynical about this. Of what benefit is this to the Perl
> community and language?

Yeesh. You'd think we were proposing a serious piece of work here! Of
what benefit to the Perl community is your staunch oposition to defining
an API that most of us want?

Is it just that you lack the imagination to see the potential benefit
here?

-sam

From: David Grove
Bart Lateur <bart.lateur@skynet.be> wrote:

> On Sun, 17 Dec 2000 14:11:50 -0700 (MST), Nathan Torkington wrote:
>
> >I think the problems with this that were raised in the past are:
> > * parsing partial source
> > * does this mean that the parser has to reparse the whole sourcefile
> > every time you type a character?
>
> Hold it. I don't think that is very practical, and for more than one
> reason.

[snip]

> But, the gist of this post is: we don't want to loose the usefulness of
> the syntax highlighter, as soon as there is one syntax error in the
> script, because this will be the normal situation while editing source.
> Parsers are generally very bad at parsing erroneous code.

You're forgetting something. Any such editor would have to be written
either in Perl, or in C with builtin Perl, in order to gain access to this
type of parser feedback. That or we'd have to communicate with perl
somehow, or the parser somehow. With that there's a huge problem: speed.

A full-fledged editor in Perl is very, very slow. That's how CodeMagic
started out. Perl had to talk to Tk (or whatever) which slows us down in
the first place. Add perl interpreter hooks into the mix, you get
something unusable on anything less than a TBird 1G. Finding keywords is
one thing, that's easy and fairly quick. However, finding structures
(matching quotes, /'s, parens, braces, brackets, and understanding perl
grammar is quite a different story. The several layers of IPC got in the
way to the extent that I had to externalize it completely.

I don't think this is a parser issue at all, unless you want to parse a
file during composition. That provides an interesting new set of
possibilities, but you still have speed issues nonetheless.

p

From: David Grove
Andy Dougherty <doughera@lafayette.edu> wrote:

> The issues of 'use Python' or 'use Pythonish' are a quite different
issue.
> I don't think anyone believes it ought to be easy to *write* the
Pythonish
> module.

I do.

That's the problem. This is a nearly ubiquitously desired objective
(writing the modules or whatever they are), but I have the fear that
actually writing one will be so undaunting that it will be a seldom-used
feature, or one that will be as often avoided as complex perl OOP (beyond
the basics), provided only by the highest levels of perl mastery. I don't
see these little languages as Perl features, but as programmer features,
in a manner of speaking. IOW, something that people in general can deal
with, without gobs and gobs of perlguts. I would imagine that even
die-hard SCO-UNIX admins would be frustrated out of using the features if
they had to read 190 pages of parser manual before they could add "use
switch;".

p

From: David Grove
Andy Dougherty <doughera@lafayette.edu> wrote:

> > That sounds too complex for what seems like a more simple solution.
When
> > you say "turn simple 'languages' into perl", that's what Dan's told
me is
> > my source filter.
>
> Correct. perl-byacc is a source filter. It takes in yacc source and
> outputs perl code. It just may not be what folks first think of as a
> "source filter" in the sense of using the Filter:: modules. For
> some problems, it's certainly too complex. For others it might not
> be.
>
> Not all problems need be shoe-horned into the same solution box. That's
> all I'm trying to say.

I _think_ we're on the same page.

The source-filter concept that I've worked on is quite a bit more than a
slough of regex substitutions. It maps grammar to grammar, or lexeme to
lexeme, or keyword to keyword, or keyword combination to keyword
combination, but needn't map an entire grammar to an entire grammar.

If we have more than one way of doing this, that's a good thing. As long
as we don't try to become a full fledged C++ interpreter, or go out of our
way to please the full c++/c/java/etc/etc language spec. It seems
sufficient to suit the goals of this task as I understand them that we be
able to output either pure perl (filter) or source tree (perl lex/yacc)
from modulated perl. Experimentation beyond that is almost
incomprehensible at this point, though admittedly not impossible (though
admittedly "why?"). ;-))

Now, for argument's sake, I'll admit there's another side. A filter won't
debug right. What we see in the debugger won't be the code that we wrote.
If we exec() it, I'm not sure precisely what we'll see. That's a strong
argument, without a solution (yet), in favor of allowing parsing rather
than mere filtering. My greatest concern is for the end user who will want
to find a quick way to add a switch statement if he wants one. The grammar
of the perl-lexer shouldn't be lex or a param-file lex-file lookalike, but
straight Perl, for their sake. From their point of view, we need to
provide them with perfect and/or pretty KISS.

p

From: Nathan Torkington Another point re: the parsing process. I think it should be possible
for any of the steps or extensions to be defined as C code or as Perl
code. We're already shooting to have C subs isomorphic with Perl subs
so this shouldn't be a problem, just something to keep in mind.

Nat

From: Chaim Frenkel >>>>> "NC" == Nicholas Clark <nick@ccl4.org> writes:

NC> I'm assuming that for performance reasons a parser-beast written in C would
NC> have code to do this conditionally compiled (like the -DDEBUGGING stuff)
NC> so that serious production perl wouldn't have the slowdown, but the perl
NC> you embed in your favourite editor would. It could even tell you the
NC> precise character (rather than line number) that it barfed on. It could
NC> run perl -c on your script without it ever leaving your editor's buffer.

NC> Not sure how you get such information back out from the API, but I suspect
NC> that making it possible to do this would need some new hooks in the API.

If I understood some of the earlier discussion. All of the
intermediate pieces/work product are supposed to be well defined
(i.e. part of the API)

So having line and offset information available in the output could be
made available.

<chaim>
--
Chaim Frenkel Nonlinear Knowledge, Inc.
chaimf@pobox.com +1-718-236-0183

From: David Grove
Sam Tregar <sam@tregar.com> wrote:

> On Sun, 17 Dec 2000, David Grove wrote:
>
> > Ok, my C's rather rusty, but are we interested in parsing that?
>
> Yes. I've heard people talk about a C frontend. Will it ever see the
> light? I don't know. Does it matter? I don't think so.

Sorry, Sam, my strong cynicism (realism ?) is not directed at anyone
specifically. The term "little language" has been redefined, and, taking
all other posts and goals into consideration, I'm responding to that
collective voice.

p

From: Andy Dougherty On Sun, 17 Dec 2000, Dan Sugalski wrote:

> At 02:24 PM 12/17/00 -0500, Sam Tregar wrote:

> >It comes down to what is meant by "little language". When I hear that
> >term I immediately think Scheme and TCL.

> For my part, at least, I've been thinking of something either LISP-ish or
> very simple parameter setting/checking (like stuff in, say, your average
> .rc file with a little control flow thrown in) when it's brought up.

That's typically what I think of, too. When perl6.0 is released, it would
be nice if it were reasonably easy to write the appropriate modules (or
whatever) and use such little languages fairly easily.

The issues of 'use Python' or 'use Pythonish' are a quite different issue.
I don't think anyone believes it ought to be easy to *write* the Pythonish
module. But it ought to be *possible*.

In between, there is a continuous spectrum of possibilities, and it may
sometimes be appropriate to call on other tools to help out.

But I don't think this should affect perl6-internals-api-parser very much.
If the parser is sufficiently clever to parse something as complex as
Perl, then it can probably be coerced into parsing something much simpler.

So let's get on with the Perl6 parser.

--
Andy Dougherty doughera@lafayette.edu
Dept. of Physics
Lafayette College, Easton PA 18042

From: Bart Lateur On Sun, 17 Dec 2000 14:11:50 -0700 (MST), Nathan Torkington wrote:

>I think the problems with this that were raised in the past are:
> * parsing partial source
> * does this mean that the parser has to reparse the whole sourcefile
> every time you type a character?

Hold it. I don't think that is very practical, and for more than one
reason.

For the editor I'm currently using, the Perl syntax coloring is a bit
simple minded, in that it does its syntax highlighting on a line by line
basis. Yes that does imply that multiline statements and strings won't
be colored properly, but otherwise, it behacves nicely. Why? because,
when I'm working on source, it is very unreasonable to expect that the
source will be a valid Perl program after every single keystroke. So no,
while editing, it is *impossible* to do proper syntax highlighting for
the whole script at all times.

More specifically, I can't be 100% sure that the lines *above* the one
I'm currently editing, are indeed valid Perl. And I don't care, not at
this moment. I definitely do not want these lines above to screw with
the syntax highlighting of the line that I'm currently editing.

So what do I envision? I think we need some sentinels, marker points in
the source, saying: "OK, do not reparse anything above this marker".
What kind of marker? Block ends, or semi-colons, those seem like
reasonable candidates.

We should also know where to stop. Do we indeed want, as soon as we type
one quote, have everything on the right of and below it, to change color
into "quoted string" mode, up until the next quote? Is this a tool, or a
light show?

If we limit the range of what the syntax highlighter affects at any
time, we'll save on processing time too. For example, we can
re-highlight the lines below the one we just edited, as soon as the
cursor leaves the line.

But, the gist of this post is: we don't want to loose the usefulness of
the syntax highlighter, as soon as there is one syntax error in the
script, because this will be the normal situation while editing source.
Parsers are generally very bad at parsing erroneous code.

--
Bart.

From: Andy Dougherty On Sun, 17 Dec 2000, David Grove wrote:

> > "Little languages", on the other hand, are a somewhat different matter.
> > They will presumably be not-so-complex and hence won't require such
> deep
> > hooks, and some redundancy there won't be such a big problem.
>
> I'm not sure how this is incongruent with your first paragraph. Could you
> elaborate a bit?

You had expressed concern about redundancy if each parser had to
re-implement a whole slew of stuff:

p> a prefilter of some sort, or multiple parsers (or worse, multiple
p> "parser/lexer/tokenizer single-entity parts"... meaning redundant
p> duplication of extra effort over and over again repeatedly).

I was just trying to say that for many small tasks, I would guess that
most of the default actions would not need to be overridden, and hence
there would not, in practice, be all that much redundancy.

> > Another route to keep in mind is spending effort working on and with
> > things such as perl-byacc

> That sounds too complex for what seems like a more simple solution. When
> you say "turn simple 'languages' into perl", that's what Dan's told me is
> my source filter.

Correct. perl-byacc is a source filter. It takes in yacc source and
outputs perl code. It just may not be what folks first think of as a
"source filter" in the sense of using the Filter:: modules. For
some problems, it's certainly too complex. For others it might not
be.

Not all problems need be shoe-horned into the same solution box. That's
all I'm trying to say.

--
Andy Dougherty doughera@lafayette.edu
Dept. of Physics
Lafayette College, Easton PA 18042

From: David Grove
Sam Tregar <sam@tregar.com> wrote:

> possible, right? Are you saying you don't think we should make it
> possible for someone to write a C parser for Perl?

For the full language spec, I don't think it's attainable, and honestly
don't see the reason for it within the context of Perl. It doesn't make
sense to me that we would want Perl6 to have the ability (though not
necessarily do it, just the ability to do it if someone has the notion) to
_replace_ python, gnu c, java sdk, lisp, etc. (That's what I'm seeing
here.) There are already languages and interpreters that do that. It makes
sense that we have the ability to change the way we write _Perl_ _within_
_a_ _Perl_ _context_ _and_ _for_ _Perl_ _purposes_, but not that we
actually become a replacement of those language compilers or interpreters,
or attempt to give the ability to replace those interpreters. It's a lofty
goal, but I can't imagine why we'd go that far, or want to, or that we
could think that we could finish that in a stable Perl6 within a
reasonable timeline. It seeks not to revolutionize the Perl language, but
the entire programming universe. I write in Python and end up with Java
bytecode... ummm, huh?

I have to be cynical about this. Of what benefit is this to the Perl
community and language? Changing how we write Perl to include different
flavors and styles... or even different languages: that's realistic; but
it certainly seems beyond the scope or desires of Perl to serve the
purposes of those languages outside of a Perl context, and the current
goals as expressed do seem to breech the confines of that context.

p

From: Nicholas Clark On Sun, Dec 17, 2000 at 02:11:50PM -0700, Nathan Torkington wrote:
> Nicholas Clark writes:
> > Would it be sane to get the parser to return suitable information on the
> > source to let a syntax analyser (such as a highlighting editor) know that
> > character positions 5123 to 5146 are a qq() string (So it can change the
> > font or the colour or whatever)
>
> I think the problems with this that were raised in the past are:
> * parsing partial source

If I understood Simon's suggestion, making the parser take truncated
"source" (ie start to arbitrary point in the "middle") was as simple as
changing the tokeniser's reaction to end-of-file

When I was thinking about this for prompting the user as part of multi-line
entry to an interactive perl shell, I wasn't thinking that the shell should
concatenate the lines of the script and send progressively longer
start-to-middle scripts until the parser is happy that "middle" is legal as
end. Instead, I was actually assuming that the parser would return a
complete opaque parser-state object, plus a hint of what what expected (if
asked, for the interactive user) and that next time round the parser is
re-started with its state object and only the next bit of script.

> * does this mean that the parser has to reparse the whole sourcefile
> every time you type a character?

I think if the parser can do the above (take an opaque state structure,
but hold no internal state) and perl provides API calls to copy the state
structure (and destroy copies) you don't need to reparse the whole
sourcefile.

Instead, the editor (say) parses tha partial source file to some point (say
just above the file position visible in the user's screen). It holds onto
this partial state (partial-state-to-known-good-point), makes a copy, and
feeds this copy back to the parser along with just the partial source
visible on the user's screen, which the editor then uses for its display.
If the user types a character (or some small localised edit) the editor
just takes another copy of the partial-state-to-known-good-point and
re-runs the parser with that and the (now changed) visible source.
If the user moves outside this region, the editor has to do the hard work
of finding another known good point (downwards is easier - just carry on
and save a later point. upwards, and you have to start from the beginning)

Of course, things all grind much more and this doesn't work if you have
two views of the same file open and are editing the earlier part of the file
in one, whilst expecting the view of the later part to update in real time.

But I think it would work providing

0: The parser is fast. (how fast is perl5's (absolute rather than relative?))
1: The parser can be fed incremental units of source
2: The stores no state internally, and can encapsulate all its state into
something it can hand back to you
3: Copying the state object is fast compared with re-parsing the script

> I think the better solution is to make Perl just a little more regular
> (as suggested in RFCs about making m in matches mandatory) so that it
> becomes easier to parse. Of course with
>
> $a = \"foo";
>
> your text editor needs to learn a little more about Perl :-) Or you

it's emacs (I will now alienate >50% of readers who will say "but you
brought it on your self." :-) Which was why I avoided mentioning which
editor it was). it's written in Lisp. Which here is :-( became I understand
about enough Lisp to write a .emacs file. If you've got an embedded perl
parser to syntax highlight your perl, you might as well put the rest in and
let it be used as another editor macro language.

> could write your own little language, so that you would write:
>
> $a = reference_to("foo");

it's actually happy with \("foo")
There's more than one way to do it.

> Or, better yet, simply write in unambiguous XML:
>
> <operator name="=">
> <operand position=0>
> <variable type="scalar">a</variable>
> </operand>
> <operand position=1>
> <operator name="reference_to">
> <operand position=0>
> <constant type="string">foo</constant>
> </operand>
> </operator>
> </operand>
> </operator>
>
> :-)

But as you knew when you suggested it, neither of those last 2 suggestions
are perl for that fluent native speaker. :-(

Nicholas Clark

From: Simon Cozens This is the fourth time I've sent this mail to perl6-internals-api-parser,
but it doesn't seem to be arriving. None of my other mail is affected, and
perl5-porters is, for once, behaving itself; why this list in particular?

----- Forwarded message from Simon Cozens <simon@cozens.net> -----

Damn this is annoying. Is it perl.org that's dropping mail or me?

----- Forwarded message from Simon Cozens <simon@cozens.net> -----
On Sat, Dec 16, 2000 at 08:09:23PM +0000, David Grove wrote:
> Thinking of just the parser as a single entity seems to me to be headed into
> trouble unless we can define in advance what type of role these dialects
> will play in the language, and at what point they merge into a single entity
> and how.

I can understand each word in this sentence, but put together they don't
appear to make much sense.

I think you're getting needlessly hung up on this idea of "dialects", whatever
you seem to believe they are. We're not parsing dialects, we're parsing
*COMPLETELY DIFFERENT THINGS*.

Python is not a dialect of Perl.

There are a number of ways we could do this. We could allow the user to use
source filters to turn Python into Perl, which is what happens currently, with
some success. We could allow the user to write their own parser and turn
Python into an op tree, which allows much greater flexibility. Or, we could
allow the user to override parts of the parser's operation, allowing for ease
of modification. Or all three.

> (or worse, multiple "parser/lexer/tokenizer single-entity parts"...
> meaning redundant duplication of extra effort over and over again
> repeatedly).

Huh? I'm just thinking of a system of callbacks. You can overload operators in
Perl, and while this is slightly confusing, it isn't earth-shattering. Now,
I'm hoping that you'll be able to overload parser operations in Perl 6.

----- End forwarded message -----
--
>Almost any animal is capable learning a stimulus/response association,
>given enough repetition.
Experimental observation suggests that this isn't true if double-clicking
is involved. - Lionel, Malcolm Ray, asr.

----- End forwarded message -----
--
Sigh. I like to think it's just the Linux people who want to be on
the "leading edge" so bad they walk right off the precipice.
(Craig E. Groeschel)

From: Simon Cozens Damn this is annoying. Is it perl.org that's dropping mail or me?

----- Forwarded message from Simon Cozens <simon@cozens.net> -----
On Sat, Dec 16, 2000 at 08:09:23PM +0000, David Grove wrote:
> Thinking of just the parser as a single entity seems to me to be headed into
> trouble unless we can define in advance what type of role these dialects
> will play in the language, and at what point they merge into a single entity
> and how.

I can understand each word in this sentence, but put together they don't
appear to make much sense.

I think you're getting needlessly hung up on this idea of "dialects", whatever
you seem to believe they are. We're not parsing dialects, we're parsing
*COMPLETELY DIFFERENT THINGS*.

Python is not a dialect of Perl.

There are a number of ways we could do this. We could allow the user to use
source filters to turn Python into Perl, which is what happens currently, with
some success. We could allow the user to write their own parser and turn
Python into an op tree, which allows much greater flexibility. Or, we could
allow the user to override parts of the parser's operation, allowing for ease
of modification. Or all three.

> (or worse, multiple "parser/lexer/tokenizer single-entity parts"...
> meaning redundant duplication of extra effort over and over again
> repeatedly).

Huh? I'm just thinking of a system of callbacks. You can overload operators in
Perl, and while this is slightly confusing, it isn't earth-shattering. Now,
I'm hoping that you'll be able to overload parser operations in Perl 6.

----- End forwarded message -----
--
>Almost any animal is capable learning a stimulus/response association,
>given enough repetition.
Experimental observation suggests that this isn't true if double-clicking
is involved. - Lionel, Malcolm Ray, asr.

From: Sam Tregar On Sun, 17 Dec 2000, David Grove wrote:

> Ok, my C's rather rusty, but are we interested in parsing that?

Yes. I've heard people talk about a C frontend. Will it ever see the
light? I don't know. Does it matter? I don't think so.

> Is Perl going to provide API to access pointers through source
> code?

Perl 6 wouldn't be the first to do so for Perl if it did. Check out SWIG.
I'm guessing that's not the route that a C frontend for Perl would take
but who knows!

> The point is, to accomplish the goal the way it's being expressed
> would be to provide the ability to do anything with perl internals
> that can be done in any other language that has ever existed, and
> leave the flexibility to do everything that every other language will
> ever do in the future.

Actually, we're just talking about allowing people to write parsers for
other syntaxes that produce Perl 6 syntax trees. Wether sufficient
skill and interest exists to build such a beast for any given syntax is
another question.

The kind of stuff you're interested in - mild twists on the Perl syntax -
will also be possible, hopefully easy. Easy things easy, hard things
possible, right? Are you saying you don't think we should make it
possible for someone to write a C parser for Perl?

-sam

From: David Grove
Sam Tregar <sam@tregar.com> wrote:

> On Sun, 17 Dec 2000, Dan Sugalski wrote:
>
> > For my part, at least, I've been thinking of something either
LISP-ish
> > or very simple parameter setting/checking (like stuff in, say, your
> > average .rc file with a little control flow thrown in) when it's
> > brought up. Occasionally things FORTHish, but only when I really
> > desperately need sleep. :)
>
> Sounds like Scheme and TCL to me. Ok, it's a bit of stretch from .rc
to
> TCL but not too far!
>
> > It is sort of a language design issue, but in many ways it depends on
> what
> > sort of facilities we can provide with reasonable cost.
>
> I really don't see why we can't provide hooks to be used by entirely
> separate parsers. To my mind it's *more* expensive to try to support
> other syntax's in the main parser than it is to just call out to
> external per-syntax parsers. Sure, the person doing the parser for a
> given language will have to write a new parser, but is there really
much
> hope that we're going to relieve them of that task?
>
> > Nope, not bytecode--a syntax tree. (Though it could go the whole
bytecode
> > route if it wants, I suppose) Going full bytecode's not necessary.
>
> Good point.


char *foo = "Some rather large ASCIIZ\nstring that's longer than this";
char *i, *j;
for(i = foo; *i; i++)
{
if(*i == 13)
{
for(j = i+1; *j; j++)
{
if((*j){j - i = *j}
}
j-i = *j;
}
}

Ok, my C's rather rusty, but are we interested in parsing that? In being
able to? Is Perl going to provide API to access pointers through source
code? Pointers are an example, not a point. The point is, to accomplish
the goal the way it's being expressed would be to provide the ability to
do anything with perl internals that can be done in any other language
that has ever existed, and leave the flexibility to do everything that
every other language will ever do in the future.

The point is, again, do we want the languages, or just to be able to steal
this and that syntax if the desire arises?

So, we want people to use perl to take in Java source code and spit out
Java bytecode (that's been expressed as one among many desired output
formats). I thought Sun was handling that particular task... but ok.

p

From: Nathan Torkington Nicholas Clark writes:
> Would it be sane to get the parser to return suitable information on the
> source to let a syntax analyser (such as a highlighting editor) know that
> character positions 5123 to 5146 are a qq() string (So it can change the
> font or the colour or whatever)

I think the problems with this that were raised in the past are:
* parsing partial source
* does this mean that the parser has to reparse the whole sourcefile
every time you type a character?

I think the better solution is to make Perl just a little more regular
(as suggested in RFCs about making m in matches mandatory) so that it
becomes easier to parse. Of course with

$a = \"foo";

your text editor needs to learn a little more about Perl :-) Or you
could write your own little language, so that you would write:

$a = reference_to("foo");

Or, better yet, simply write in unambiguous XML:

<operator name="=">
<operand position=0>
<variable type="scalar">a</variable>
</operand>
<operand position=1>
<operator name="reference_to">
<operand position=0>
<constant type="string">foo</constant>
</operand>
</operator>
</operand>
</operator>

:-)

Nat

From: Nathan Torkington Andy Dougherty writes:
> Now matter how we slice it, it's going to be very hard for the first
> person to twist perl6 to parse something that is both complex and very
> different from Perl6. And I'm unconvinced that this difficulty ought to
> hold up the entire process. It would be quite ironic if perl6 never gets
> off the ground because we can't figure out how to make 'use Java;' easy.
>
> "Little languages", on the other hand, are a somewhat different matter.
> They will presumably be not-so-complex and hence won't require such deep
> hooks, and some redundancy there won't be such a big problem.

Here are the steps I see:

* Data source: string, SV, file, whatever.

* Optional textual filter:
* takes: the data_source object
* emits: another data_source object

* Lexer with optional extensions:
* takes: data_source object
* emits: token_stream object

* Parser with optional extensions:
* takes: token_stream object
* emits: parse_tree object

* Tree optimizer:
* takes: parse_tree object
* emits: parse_tree object

* Code Serializer:
* takes: parse_tree object
* emits: bytecode_stream object

* Code optimizer:
* takes: bytecode_stream object
* emits: bytecode_stream object

Perhaps another way to view it is like this:

* Data source

* Lexer
* takes: data_source
* emits: token_stream

* Parser:
* takes: token_stream
* emits: parse_tree

* Serializer:
* takes: parse_tree
* emits: bytecode_stream

Then each of these steps has pre- and post- handlers. If you wanted
to source filter, you'd do it as a lexer 'pre' handler. If you wanted
to optimize the bytecode, you'd do it as a serializer 'post' handler.

I picture each of the steps as being extensible either globally (so
you could easily have a compiler designed to only compile Python, by
swapping the lexer and parser), or lexically (so that for this file or
block, I'm writing in my little language that really compiles down to
a regular expression).

I get the feeling that I'm expecting the bytecode to say "this code
uses Python vtabled code". That is, the data-types are Pythonish not
Perlish, and the code operations acting upon them should be Pythonish
not Perlish.

Nat

From: Sam Tregar On Sun, 17 Dec 2000, Dan Sugalski wrote:

> For my part, at least, I've been thinking of something either LISP-ish
> or very simple parameter setting/checking (like stuff in, say, your
> average .rc file with a little control flow thrown in) when it's
> brought up. Occasionally things FORTHish, but only when I really
> desperately need sleep. :)

Sounds like Scheme and TCL to me. Ok, it's a bit of stretch from .rc to
TCL but not too far!

> It is sort of a language design issue, but in many ways it depends on what
> sort of facilities we can provide with reasonable cost.

I really don't see why we can't provide hooks to be used by entirely
separate parsers. To my mind it's *more* expensive to try to support
other syntax's in the main parser than it is to just call out to
external per-syntax parsers. Sure, the person doing the parser for a
given language will have to write a new parser, but is there really much
hope that we're going to relieve them of that task?

> Nope, not bytecode--a syntax tree. (Though it could go the whole bytecode
> route if it wants, I suppose) Going full bytecode's not necessary.

Good point.

-sam

From: Dan Sugalski At 12:41 PM 12/17/00 -0500, Bradley M. Kuhn wrote:
>Nicholas Clark <nick@ccl4.org> wrote:
>
> > Something I though of:
> > If you're trying to write an interactive perl inputer - either a perl shell
> > or just the command prompt on the debugger it would be useful if you
> > could tell the parser that the chunk of source you're giving it may be
> > incomplete.
>
>I really like this idea, although I am unsure of how we might implement it.
>I don't recall any of the compiler texts I have read over the years talking
>about formal methods for writing such a "partial programs are acceptable"
>parser.
>
>
>Lisp-like languages handle this, but s-expressions are so trivial to parse
>that it's no help to simply "follow" their example.

BASICs generally allow this.

The rule I was thinking was we'd take source on the fly, and execute bare
statements and (opionally) print the results. Blocks wouldn't get executed
until they were closed, and things like "sub" wouldn't get executed, only
parsed. (Like they are now)

Basically read to the end of a statement and eval the statement, only do it
with a scopeless eval, so things like lexical variables wouldn't
immediately disappear.

Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
dan@sidhe.org have teddy bears and even
teddy bears get drunk