The Forum Reader

sounds greate!

-----------------------------

DaHeR.RailgunMaster

MFC and COM, hmm. That''ll be very Mozilla/Linux friendly...

Yeah, I know, and I''m sorry about it. I don''t know enough Linux programming yet to be able to write something - but the whole thing is powered by XML, which has very strong support under Linux (as far as I can see).

The main reasons I gave up on the original chrome-based browser was not because it was dependant on Mozilla - I don''t consider that a problem - but because the Mozilla chrome:// platform isn''t powerful enough to support full applications yet. I use MSIE4 components - the actual rendering engine is available as an ActiveX component - to display stuff, by writing HTML to a ''scratch'' file and then pointing the browser component at that file; there''s nothing like that under Mozilla. It''s just not designed for full-scale apps yet.

The communication between client and server components is all done in XML via HTTP. So if someone wanted to write a Linux port (or a port for any other platform, for that matter), it wouldn''t be too difficult - I''ve tried to keep the Win32-specific stuff seperate from the rest of the code anyway, so things like internal data structures could probably be built directly under other platforms with no modifications.

There is a security issue, though. The reason I''m not doing this as an open-source project is that the less people know about the way it works, the better. The design itself is still secure - the username and password get sent with every request, and the server must authenticate them each time - but still, lack of information about a system is a basic security principle.

Meh... if the Staff were to be happy with it being made open source, I''d have no objections. I''d have to tidy up the design documents a little, though.

Superpig
- saving pigs from untimely fates
- sleeps in a ham-mock at www.thebinaryrefinery.cjb.net

Let me know, when it''s done. If it''s only the network layer, then it wouldn''t be too much work, I could port it to Linux when I find some time.

quote:

---

Original post by superpig
There is a security issue, though. The reason I''m not doing this as an open-source project is that the less people know about the way it works, the better. The design itself is still secure - the username and password get sent with every request, and the server must authenticate them each time - but still, lack of information about a system is a basic security principle.

---

Security through obscurity is fallacy. I suppose the security issues you mention are server-side? (Cos if they''re client-side, then you should open up the code and let other people have a crack at it; one of them just might have a fix). If the security issues are server side, then I''d suggest corresponding with the staff directly (or a mod, actually, since they (we) have a mailing list with the staff where such things are discussed).

How does it affect bandwidth use, though?

I think he meant client side security issues, such as people faking the GDNet server protocol and trying to connect to the client (to harvest the login password). Or trojans specifically designed to hook up to the client, going up to a point where people would try to distribute binary versions of the reader with integrated trojans (eg. in the announcement forum: "I have updated the forum reader with several new features [+trojan] ! Download here.")

A partial solution to that would be to use asymmetric encryption for the transfer of passwords and other sensitive data. The data stream should be compressed at the same time, that will minimize bandwidth. Standard images (eg. the forum icons) could be fully cached by the client.

Oh and don''t forget to include an advertisement hook/area in the reader. Otherwise, GDNet won''t be able to use it right now, due to their advertisement contracts.

Oluseyi has persuaded me that I should go over (at least) the basic way the thing works. After all, I''m just a lowly student, you guys know much more than me about security and stuff.

Upon connection to the server, a ''packing list'' file (in XML format) is retrieved. It''s about 500b, if that. It contains the following information:

• The path to the ''forum descriptor'' file (which describes how the forums are arranged into categories, names of forums, and so on), and when that file was last updated
  
• The path to the smiley file, which gives the paths to all of the smiley graphics, along with the codes required to produce each one, and the date that file was last updated
  
• The paths to the ''template'' files, which describe how the post data (username, post content, post time, and so on) should be laid out in HTML (and when they were last updated)
  
• The paths to the stylesheets required (and when they were last updated)
  
• The paths to the respective ''connection points,'' which are the files to request to perform certain actions (e.g. "viewthread.asp"). These aren''t dated.
  

The packlist is compared to an offline version, and the times given in each are compared. If a file is not listed in the offline copy, or is out of date, then it is downloaded. This means that the first time people connect, it will be slow (as they download all the smilies and templates, etc), but after that, they''ll only download files as they need them.

I''ve hooked the CHTMLView window that I use to display stuff, on the BeforeNavigate2 function - which gets called when a link is clicked. There, I pick out certain links which start with things like "profile:" or "viewthread:". So far, the profile link is the only one implemented, but it will mean that user profiles can be opened in dialog boxes and so on.

When a thread is requested, the username and password are sent (as with all actions, to allow the server to do as much validation and checking as it wants), along with the thread ID and starting post (so you can view 30 posts starting at post 30, to implement pages). The server, in response, sends back something along the lines of (angle brackets to square for this example):

Content-type: text/xml
[THREAD ID="1" TOTALPOSTS="5"]
[POST ID="4053" UID="27" UN="Superpig" PT="2002-11-03 23:07" SMI="T"]This is a post, don''t you know.[/POST]
[POST ID="4054" UID="27" UN="Superpig" PT="2002-11-03 23:08" SMI="T"]Really? Fantastic.[/POST]
[/THREAD]

I think you can see the advantage in bandwidth usage. The way things look can be customized by the templates sent from the server - even arbitrary information can be displayed. The templates are normal HTML files, with [? ?] tags in them. So, if I wanted to display the username, I would use [?un?] (as ''un'' is the attribute name in the XML). This would easily allow things such as avatars to be implemented - the XML ''post'' tag contains an ''avatar'' attribute, and then the template contains [IMG SRC=""] to display it.

I have every intention of adding more things to the list of cacheable items. Ultimately, the program ''mirrors'' the required bits from the server on the client, so provided the templates use relative URLs, then pretty much anything could be cached (forum icons included), using the same mechanism as for the smilies.

I don''t know much about encryption and so on. I thought maybe MD5 hashing would be acceptable, but it''s not quite solving the problem; I don''t know what kind of encryption is used for the current forums, but no doubt it would be easiest to use the same in the forum reader.

As for the advertising, the templates should allow for it (although, an integrated-into-the-interface ad slot could be better, a la Limewire). If there''s a fixed URL for the ad banner, then it can be written into the templates as an absolute url (which will cause it to be downloaded, not cached). And because it uses the MSIE browser component, all plugins under MSIE will be available (including, for those who have it, Flash).

Open-sourcing the project would allow modification to be made without limit. It would help to make the app more stable, but one thing I expect the most is that people would modify it to filter out the ads. What I think would be ok, though, would be to make the source available to certain people (i.e. staff and mods, and anyone else given permission by staff).

I don''t know the terms of the advertising agreement, but I don''t think the forum app would really qualify as ''part of the site.'' So would ads really be necessary? The reader should cut bandwidth usage significantly, so the ads may not be necessary (for long, at least - although, the ATI contract was a year-long thing).

Superpig
- saving pigs from untimely fates
- sleeps in a ham-mock at www.thebinaryrefinery.cjb.net

quote:

---

Original post by superpig
I don''t know the terms of the advertising agreement, but I don''t think the forum app would really qualify as ''part of the site.'' So would ads really be necessary? The reader should cut bandwidth usage significantly, so the ads may not be necessary (for long, at least - although, the ATI contract was a year-long thing).

---

Yes, ads would be needed. The other parts of the site (articles, news, etc) use very little bandwidth compared to the forums, and a good number of people (read: bandwidth users) don''t even go outside the forums.

---

This post qualifies for 100 per cent Canadian Content under the rulings of the Canadian Internet Commission and the Federal Ministry of Communication. There are four Americans who worked on this post, but they all have landed immigrant status, and have signed CRTC affidavits swearing that they drink beer, eat back bacon, drive snowmobiles and wear toques. Any resemblance between the Content of this post and the content of any American post is purely coincidental and not the intention of the poster or the various Internet Agencies of the Canadian Government who have screened these posts prior to bulk erasing in accordance with the policies of the Federal Internet Identity Board.

OK. Well, as I said, the template system would allow for banner ads (and, I think, things like the Raedon ad too - but I''m not going to talk about that right now ).

The other thing is about contracts vs. method, i.e. does gamedev get paid a fixed amount for the ad''s lifetime, or does it get paid each time the ad is displayed, or what...?

I fully expect that there''d still be a lot of people posting in the forums through a normal browser: people who aren''t using PCs for web access (such as myself), anonymous posters (if they last much longer ), people who don''t have the privileges to install the software on their machines, or people who simply don''t like the client. All those people would still be prone to normal advertising.

I guess the site is operating at a loss; I don''t know by how much. It will be possible (eventually) to calculate an approximate ratio of normal-forum bandwith to reader-app bandwidth for all common tasks (view thread, etc), and then, given bandwidth costs and a guess at the number of people who would use the client, it''d be possible to work out how much money the reader would save. But if things aren''t spiralling out of control at the moment, I''d guess that the reader will allow gamedev to start making a profit - and, eventually (I hope) remove some ads.

Superpig
- saving pigs from untimely fates
- sleeps in a ham-mock at www.thebinaryrefinery.cjb.net

quote:

---

Original post by Michalson

Yes, ads would be needed. The other parts of the site (articles, news, etc) use very little bandwidth compared to the forums, and a good number of people (read: bandwidth users) don''t even go outside the forums.

---

This from a guy with over 13,000 posts, most of them being crap (eg. Ultimate Cat Thread).

If done right, I think it could save GameDev alot of bandwidth, i.e., ask the server which topics have been changed since the last time we had used the client, and for loading a topic, only reload the posts that are new or that have been edited since that date, replace html symbols with a special character followed by a short code of 1 or 2 bytes, and since it will be mostly text, it should be able to get some impressive compression ratios.

It will also stop people from pressing F5 repeatedly while waiting for an answer for their threads.

Progress update:

The UI is fully functional (as you would expect from MFC). The client connects to a server (on my localhost), retrieves the packlist file, parses it, compares it to the offline file, and downloads definitions and smilies as appropriate (templates, stylesheets, etc to come later). Also, I''ve tested the basic hyperlink-hooking mechanism, and it works great.

And for anyone who''s wondering, yes I have actually written a small forum package that I''m using as a server for all this.

Other than the coolness factor, though, how is this better than converting GameDev''s forums into mailing lists (which would let you subscribe on a per-forum basis)? I guess for those with size limitations on their mailboxes (webmail) it''s not such a good deal...

Seriously, though, I think that the mailing list option is a way to reduce server load tremendously without building a whole lot of new technology. The site would hold archives by forum/topic/date, articles, news and special activities (like ascii fishtank competitions). Infrequent visitors could receive daily digests for specific forums with links to the thread archive. We could even let people still post via the site.

Is this feasible, or am I completely off?

nice stuff, can''t wait to see this program.

---

My compiler generates one error message: "does not compile."