Lois & Clark Forums Forums Fan Fiction Fanfic Related Fanfic in epub formats - archive anywhere?

Hi, Doranwen,

I really like your idea of making epub versions of stories available. I've been reading fanfic as epubs too, using Stanza on my iPod Touch. I second everything Lynn S.M. says about Stanza. Fantastic app.

Your description of Sigil is intriguing. Just tried it. Wow! With the fields for editing metadata (title, author), it's nicer than desktop Stanza at creating epubs, in my limited experience -- but Stanza is aces as a reader.

Regarding your problem with line breaks, there's the Word solution others have described. You could also give the Formatter page on the archive a try. It converts among fluidly wrapped text (as in your word processor), archive-formatted text and simple HTML. You can strip out the extra breaks using it. It'll work on stories using our current format, which has a blank line between paragraphs. For older stories that use tabs but no blank lines between paragraphs ... not so great.

Though when I opened an archive text file in Sigil, it knew what to do -- just like Stanza, it stripped the extra line breaks, and all paragraphs were fluidly word-wrapping. AND it did it for both old and new stories with their varied formatting. So, no tedious manual line editing or search/replace routines required. It thought the header block was one paragraph (like my formatter script does), but you can easily "press Enter" to restore the line breaks you do want.

Both Stanza and Sigil fail at poetry formatting -- if the lines of a poem's stanzas are single-spaced, they get word-wrapped into one paragraph. So poetry and song lyrics would have to be manually edited, just like the story header block.

There is one really nice feature desktop Stanza offers over Sigil: It lets you export a story in various flavors -- HTML, Word, plain text and PDF -- in addition to epub. It can even export to the mobipocket format, which Amazon's Kindle supports. (The Kindle, probably the most popular ebook reader of all, doesn't support epub, strangely enough.) Looks like Sigil saves as just epub. You might try using desktop Stanza for your intermediate conversions.

While I think it would be great to host epubs on the archive, the logistics will take some working out. Short-term, maybe a file uploader tool to let you upload your epub files to the server? Then maybe a wiki page you could edit in order to create download links? That would make them available to readers fastest. It sounds like bobbart and maybe others would like to contribute too. (Thanks, bobbart!) This could be a workable solution for all, though it may take some coordination to prevent duplicate stories being posted.

Long-term, we could put the epub download links next to the main story links in all the catalog pages. That'd take some time to pull off.

For a sample of what this could look like, see the archive's Filename Z page. As one of the shortest catalog pages, it makes a great sandbox for testing. Aren't the icons lovely? Just added the epub versions courtesy of Sigil -- they're the black buttons at the right of story titles. (Firefox is trying to open them if you left-click on them, but right-clicking and "saving as" works to download them to your hard drive.)

Filename Z is a prototype of what I see as the future of the archive: The site would be database-driven, with HTML the default story format, and text, OpenOffice, PDF and epub flavors (thanks to your suggestion) also offered. Maybe mobipocket too. But imagine nicer icons for the different story formats.

My nebulous hope is that if we feed a word processor story file into the future database, a magic conversion process will kick in, creating all these different formats. Getting to the magic part is what has me stumped. I've been plodding through PHP books, trying to wrap my English-major brain around programming logic. But it seems doable. OpenOffice format, for example, is basically a zipped XML file. Couldn't styles designated in an OpenOffice template match up with CSS styles for HTML pages, and even for epub docs too? Headline tag to headline tag, author tag to author tag, etc. And tools exist in PHP to manipulate XML documents and convert them to PDF and other formats. There's got to be a way, without doing it all by hand. If a story is properly tagged going into conversion, it ought to work. (She said blithely.)

An automatically converted epub, if that's even possible, would be missing the value-added features only a human could provide, though -- the chapter breaks, TOC and appropriate poetry/lyrics formatting. A bonus of going database, besides more powerful searching, is eliminating hand-coding. I suppose some will always be necessary. It's all hand-coded now.

Sorry, got off-topic.

Permissions in that authors' stories will be converted into a format they didn't anticipate, or permissions in that you'd like to host the files elsewhere?

I'm perfectly happy for us to host epubs on the archive, and I'm equally as happy if you'd prefer to post them elsewhere and have us offer a link to your page. But I'm fuzzy on if there's an implied agreement between the archive and authors about hosting their stories. When they submit a story to the archive, they know it's going on the archive. Would they be upset if it went other places too? Authors who happen to be reading this, would you object? And how about story formats: Would you mind if your story were turned into HTML, PDF, OpenOffice and epub?

A precedent may have been set for this years ago. Soon after Rhen's collection of FTP story files turned into a Web-based archive, someone out there in the ether created a mirror site. An Australian guy, I think. Nobody objected as I recall. Haven't heard anything of it in the last 10 years, so it's probably gone.

But I do remember we lost some authors and their stories in the process of going from a sheltered FTP site to a very public Web site. Because they were using their real names, they felt exposed. And for one reason or another, they were unwilling to adopt a pseudonym. One author -- Gail L. -- wrote a great story about Lois and Clark being stuck in an elevator for a few hours. I lost it due to a hard drive crash, darn it, along with all her other stories. Not that this last relates much to the issue at hand, other than that people sometimes offer objections you don't anticipate. (And I really miss stories when authors pull them off the archive.)

In the case of audiofics, as you say, it's a bigger issue. There was a thread where authors opted in to give their consent to have their stories recorded. But epubs are not that radical of a transformation -- it's still text you read on a screen, though more attractively formatted.

I'd say permissions issues are LabRat's call. (Tossing it back to you, LabRat! :p )

Thank you, Doranwen, for poking your head back in. I think it's an exciting idea, and clearly there's lots of enthusiasm.

And thank you too, bobbart, for your interest and the Stanza how-to.

So what's the next step? I can start looking into a file-upload script and prepping a wiki page if that's agreeable. Or we can talk more and explore other options. I think it'd be good to hear LabRat's thoughts and see what our authors have to say.

Best wishes,

Lauren

P.S. I had to go to Wikipedia to find out who Rodney McKay is. Thank you for the comparison, LabRat, but I fall way short! He's probably more like bobbart, who can use a binary hex editor. How ubercool is that?

Great post, Lauren. I didn't know about the different formats on the Archive. There's been a poll for a long time on the Archive asking if people want stories in HTML. Maybe there's an indication there of interest.
I have no objection to my stories, few that they be currently, being posted in other formats. I suspect that if someone from this board posted them elsewhere, I would have no objection either.
But I went to FilenameZ like you suggested and saw it in PDF. I'm in hog heaven now. PDF goes directly to my reader and works perfectly, so I'm happy with that. PDF has the advantage that no receiver of the file can make changes to it. Though why someone would want to change the stories, I have no idea. We get a lot of secure mail (travel reservations) in PDF.

Thanks for all your hard work over the years, Lauren. And all this from an English major! I'm impressed.
much regards
Artemis

Hi Lauren! Your vision for the Archive vNext looks great.

The whole talk about automatic document conversion certainly causes some PTSS flashbacks over here Fortunately for us, novel-type text is quite simple, as far as formatting goes. So, yeah, as soon as you have the document broken up into paragraphs, the rest should be quite doable. At least in C#/.NET.

Michael

It also has the disadvantage that it's designed to display things exactly like the original. This is not a good thing when it comes to e-book readers, since their screens are considerably different sizes than a computer screen (the Astak does a fairly good job at text reflow on pdfs, but it's just making the best of a badly suited format). PDFs are considered one of the very worst formats when it comes to e-book readers. On a computer, on the other hand . . . I love PDFs! So it really depends on the purpose. Having a variety of formats is good for that reason.

I avoided the Kindle and nook for reasons precisely like that--I wanted lots of options for file formats, not just one or two. For small files, I don't have to convert everything to a specific file format, which is really nice. For the big stuff, I can either use Calibre for a quick convert if I'm in a hurry, or paste into Sigil for the nicer look.

Sigil IS limited in its output format, but it was only designed as an epub editor, because there wasn't anything out there easy enough for newbies yet powerful & featured enough to produce quality files. Since epub is the best format for the Astak, I figure I'll just create that when I do need to convert.

I just loaded the PDF file for "The Zipper" on my pocket Sony Reader (vertical read, screen 3" wide) and it wrapped perfectly with no dangling short sentence ends. I'm guessing the PDF format doesn't have the hard stops of the Word files.
I even increased text size and worked perfectly then too. I was indistinguishable from an epub doc.
So I continue to be happy. If Lauren continues with the demonstrated programs of Filename Z, I'm good to go with that.
regards
Artemis

Sounds like yes, the PDF wasn't formatted with the hard stops, AND your Sony Reader's software has good text reflow ability. Without either one of those two, PDF is a nightmare on e-book readers. Epub, on the other hand, is a standard e-book format designed for e-book readers. Don't get me wrong, I'm glad there's PDF versions too! But not every PDF works well on my e-book reader, so I'm glad I can create epubs.

For my part I'm all in favor of multiple formats. I'm all for making the documents as accessible as possible. I'll be happy to help and when we get going I'll certainly do what I can. I'm happy to have my own works change format. It's the words and ideas, not the delivery mechanism.

I love the 'Z' page. It would be great to have something like that everywhere. However, my suggestion would be to start with all the Kerth stories. If we have to start somewhere, why not there? However, that's just an idea. I'm in favor of whatever we can do.

As for this: WOW! I'm mentioned in the same paragraph as Dr. McKay. Using a hex editor is just a sign that I'm an old-school assembly language programmer. Still... WOW!

Oh, one more thing. In my outline earlier I forgot one important step.

1.5 As soon as you load the file into MS Word, edit the Title and Author in the Properties field. Stanza will pick these up when later when you load the .doc file and you will get your Author and Title exactly as you entered them.

Bob

We don't have consensus on what our procedures will be with epubs going forward, but I've found a file upload script to use for getting epubs on the site. Doranwen, bobbart, it's ready for use if we're ready to start collecting them. Thank you for volunteering your time.

Artemis wrote:

That's great to hear! Hoping other authors will feel the same.

HTML looked to be winning toward the end. It seems inevitable. The stumbling block toward that effort is the sheer number of stories there are to convert. Lots to learn in getting that done. The only place where extra formats are available for stories right now is the Filename Z page. I figured OpenOffice for flexible formatting/printing, PDF for quick printing, and plain text because that's our legacy format. But this is the year of the e-book, so epub is a natural, and a great suggestion from Doranwen!

I'm glad it worked for you, Artemis! I didn't know any e-readers could do flexible word-wrapping with PDF.

Have you made your own PDFs? If not, I highly recommend it. OpenOffice makes it a piece of cake! The PDF files I posted on the Z page were made using OpenOffice (free, open-source, Microsoft Word-compatible, and available here in Windows, Mac and Linux flavors). Just a few simple steps:

1) Use the archive Formatter to convert a story from archive text into fluidly wrapped text.
2) Paste the output into OpenOffice Writer, then clean up the header by restoring the line breaks after title, author name, etc.
3) Do whatever formatting you like. (If you're creating the PDF for printing, you might adjust the font styles and sizes, margins, indentation, format the title as a heading and format the story in columns.)
4) Click File/Export PDF.

I was figuring PDF would be used primarily for printing and tried some different layouts. The PDF for Paul's "Zombies" story is in two columns, magazine-style. I don't know how that one would flow in your e-reader, but two columns seems to be an efficient way to print.

I have visions of filling three-ring binders with fanfic. Would be really nice to have a printer that could print on both sides of a sheet of paper. But Doranwen's print-on-demand idea is even better to dream about. Real, bound books! I'd start clearing shelf space now. The complete and collected works of [insert author name here]. Sigh.

Thanks! Trying to learn programming is maddening.

Darth Michael wrote:

Thanks!

Argh! Sorry about the PTSS. I keep saying to myself, "It should be simple!" But I keep stumbling around in the dark. You sound like a guy with a book of matches. C#/.NET? That stuff looks scary.

bobbart wrote:

Thanks, bobbart. That's a beautiful way to say it.

I think anywhere people would want to start is good.

You're an old-school assembly language programmer? That stuff looks really scary!

Best wishes,

Lauren

P.S. I was working under a bogus assumption about Stanza when I typed this in my previous opus...

The first part is true -- Stanza does strip the extra line breaks from archive format text files, but only for any epubs you save. I assumed that would also hold true for the Word and PDF files you produce using it. It doesn't. Drat. PDFs and Word files saved from Stanza keep those line breaks intact.

But hey, there's still the archive Formatter.

Word 2007 on System 7 on my PC has PDF as a choice, so I simple saved it that way after dl a story from the Archive.
Oh, the formatter is awesome, but I did one without using it and that worked fine too.
One thing I did find out that you have to pay attention to the type font when it is in the Word stage. Not changing it made it really tiny to read at the S setting, and L is too large.
BTW, we submit to the archive in text. Could we submit another copy in PDF or Word.docx directly and would that save time? It has open document as well, if that works better.
The authors could submit their own stories and that gives permission as well.
Thanks again for all your work over the years.
regards
Artemis

Well, I've already started converting the stories I do have saved offline into epub (though going back to the Archive to pull a fresh copy to work from). Sigil is not removing the hard returns so I'm dumping them into Word and doing a series of Replace Alls. Works pretty well, actually.

The one part I'm trying to figure out is just where to draw the chapter line for stories which are not on the MBs (where the Fanfic TOCs are great for seeing the section divisions the author wanted--I've got a tab here in Firefox open right now for doing those searches). I'm tackling Anybody's Baby by Annie M right now, for instance, and finding it a challenge to pick a scene to split at. She's got frequent scene breaks, so any one of them could work, but about how long should a chapter be? (Keeping in mind I'm now looking at a full-screen fic with no premature line breaks, approximately how many pages would you say?) Try to end it on a cliffhanger or suspenseful moment when possible? Might be good to get some guidelines discussed so we have an idea of how to go about it.

As for permissions, considering they've already given permission for it to go on the Archive, I can't see that a different file format would be any issue. We're not adding anything (except for "Chapter 1", "Chapter 2" as necessary) and simply re-formatting. Sounds like we're probably good to go there. So as soon as you've got things set up to do uploading of files, I'm ready to share some. A wiki sounds like a good idea, as long as it's tied to being registered here and not just public, of course. *g*

Are you considering expanding to all four formats at one time? Would you need people to do the converting manually or is there an easy way on your end to batch process the whole lot? (Or perhaps, once there's an epub, it's easier to go from that to another? I don't know of any simpler way to get rid of the hard returns than all the Replace Alls, and I have to think about what I'm doing. For instance, some fics have a space at the end of each line, some don't, and some have sections with three returns where I only want two, in addition to two returns where I only want one--epub spaces out paragraphs naturally--so I have to do at least five conversions to get everything to come out right and be mentally processing how to do them. I'm not sure there's any way to automate that!) Though it looks like the hard returns aren't really a problem with html--they do look a little odd for pdfs, though.

I agree. I don't think there's any reason for authors to object. But if anyone does, it's a simple matter to email or PM me and we'll exempt the story from the additional formats. Shouldn't be an issue, I don't think.

I like the Z page, Lauren!

It would be nice if HTML could be a format choice, too. I recall the last time it was discussed we couldn't reach a concensus on which fonts/colours to use which would enable everyone to view the story. But if it was simply another choice of format, among others, rather than the only choice, then I think that issue would be less important.

LabRat

Well, I am definitely stuck when it comes to deciding chapter breaks. Obviously if it were posted on the mbs, it's simple; just see where the author broke up the parts, and do the same, even if you have to insert a "Part Four" where there isn't a scene break. *g* That way we can be said to be following their very wishes in that respect.

But what do we do with the stories that *weren't* posted on the mbs anywhere? About what size would you break it into two parts? As soon as it hits 30 kb? 40 kb? Or wait till larger? (Obviously, a fic of 200 kb, for instance, needs to be split in places--the question is simply how many and where.) About how long should a chapter be? And if there are frequent scene breaks, what sort of break should we choose to end a chapter on?

If I should be starting a new thread for this, let me know. But I think this needs to be answered so we can do the best job possible. Authors, what guidelines would you prefer if you hadn't posted it on the mbs at all? Anyone have any thoughts on this?

Did you already check zoomway's board? It's got the older stories (dating back to spring 2001) before lcficmbs went online in 2003.

Michael

Ooh, I had forgotten about that! Thanks! That does indeed help--sure enough, Anybody's Baby is indexed there. That should help with most of them.

However, I know there are a few fics that go straight to the Archive, bypassing the MBs--there's one author who does that a lot, I think (the name is niggling at the back of my mind, but I'm having a hard time remembering). What should we do in those cases?

I try to make my chapters about 20K. Some are 30K. Yes, you try to build to a cliffhangar. One thing that will be different on the Archive is that authors often include a Previously section at the beginning of a chapter to bring the readers up to date. That should go away when you are reading a story as a whole. I edit them out when I submit to the Archive.
Good luck with your efforts. For those of us who can change them ourselves, let us know where you want them submitted. Would the Archive be up and ready to do that?
regards
Artemis

Artemis wrote:

I didn't know Word could do that. Cool!

That's an interesting idea. It's more in LabRat's territory, as well as that of the general editors. I don't know if they all use Microsoft Word, or what version. Or if they use OpenOffice , which is totally free and Microsoft-compatible. Terrific program. Had to get the plug in.

You're welcome, and thank YOU for writing stories and contributing to the archive.

LabRat wrote:

Excellent! Thanks, LabRat.

Working on it, Chief.

Doranwen wrote:

Good points, and we've got LabRat's OK. So we're good to go!

I've got an uploader ready and have installed DocuWiki on the archive. (I beat my head against the wall for a while trying to install MediaWiki, but couldn't get it past the database configuration on the archive's server.)

I'll send the links to you and bobbart first for testing. If you see problems, please let me know!

If the wiki turns out to be useful for this project, maybe we can use it to collaboratively work on other projects as well.

It's something I've been considering for a long time, except for the epub flavor -- that's new. I'm hoping to plod away some more at the batch conversion idea, eventually tying it all up with the database. (Batch conversion with the exception of epubs, pointing to the ones you guys will be creating using a new field in our future database/current spreadsheet.) Though people stepping in to help correct formatting post-conversion on the HTML/OpenOffice versions would be wonderful. But it'll be a while reaching that point.

Many moons ago I prepared an archive-to-Word how-to addressing conversion and spacing issues for the archive staff, and it's still online . Recording the steps as a macro turned the whole multi-part search/replace process into a single click. The macro is still there too , but I don't know if it'll work in the current version of Word.

You'll have better results recording all the search/replace steps you go through in your newer version of Word to create a macro more specific to your needs. Things have probably changed a lot in Word in nine years -- and I think I was even using Word 97. Just noticed mine doesn't address turning three returns into two after getting rid of the spaces before and after returns. (^p^p^p to ^p^p, repeat.) Drat.

Anyway, if you haven't, please give macros a try! Could save you tons of time and greatly streamline the process of making a story epub-ready. And if you find it does simplify the job, please consider sharing the macro.

I think it's exciting that we're doing this, and doing this as a group project.

Best wishes,

Lauren

Sending story docs between myself and the GEs, we use text - simply because it seems to throw up less compatibility issues between pcs.

But I think the GEs themselves use various formats when working with the authors. Whatever they're most comfortable with. I don't think they all use MS WORD, no.

I have no idea what would be involved in using something other than WORD or text docs, but that's no reason not to give it a go if it makes things easier for you guys.

Yes, and hasn't that saved my butt time and again over the years. Well, hours of tedious work, at least. <wistful sigh> I really miss that little guy.

LabRat

I know there are a few sites that can change the font/background colors with a drop-down menu (fanfiction.net has recently done this with a dark/light option). It's a fairly simple javascript that can be added to the HTML pages.

I don't know if space is an issue on the archive, but I was recently looking at how to make PDFs on the fly. ezPDF was one option, but I was having problems getting it formatted. The drawback is that every file is formatted the same way (single column, double column, etc). It can pull the format out of the original file, though, so if it's converting an HTML file, it can convert all of the bolds and italics.

I've considering making a macro for it (I have Word 2003 so not sure how easy those are to work with there), but the more I think about it, the more I'm not sure that would work. For instance, some stories have a space at the end of each hard line break, others will need it added when the line break is omitted; those would have to be dealt with separately, and you have to look at the story to tell which is which. Also, I've taken to adding my placeholder symbols at the beginning of the last three lines of the heading info just to save on the hassle of splitting the line later when my Replace Alls remove the line breaks, lol. It's actually quicker to add them beforehand than to go hunt down where to hit Enter later. Some stories have three ^p, others don't. And there's still the skimming through to make sure all formatting worked great--I've found some extra carriage returns at scene breaks here and there (and some scene breaks missing at least one carriage return, causing some bunching up). I do that as I break it up into sections, so it's not really extra work, but does require some monitoring.

So although I'm sure someone macro-savvy could work past those, I've got it down pretty fast so it's not too much of a pain. Three authors done and counting now . . . the only slowdown will be adding them to the wiki. *g*

Thanks for doing the work to get it set up for us to upload, Lauren! I'm glad to be able to share what I've gotten done with the rest of the FoLCs. Couldn't have imagined this could happen, the first time I started reading L&C fic!

Hey! I got one uploaded!

I hope I can be forgiven a little for loading SkyFall first. I wanted to see how a large multi-chapter story would work.

I'll try to get a few more stories done before the evening is out. (And they won't all be my own stories. )

The process to upload seemed pretty straightforward. I have many more stories in epub format but I want to go slow and make sure I've done the best job possible to get the format clean.

Thanks again, Lauren. And you too Doranwen. I'd like to go faster but time is a precious commodity these days. (Not to mention a WIP that is demanding my time.)

Bob