Lois & Clark Forums Forums Fan Fiction Fanfic Related Fanfic in epub formats - archive anywhere?

*waves hi and grins sheepishly* I'm actually poking my head in here, after I don't know how many months. I had a question I wanted to pose.

See, I got an e-book reader recently (Astak EZ Reader Pocket Pro, if anyone's curious which), and its best format for reading is epub. Although it can read txt files just fine, testing the Archive's files brought me to a screeching halt.

A small minority of fanfics on the Archive, when saved, have so many words and then a forced line break. This is easy to read on a computer screen, since my computer screen is far wider than the line is. On my e-book reader, on the other hand, the screen there is smaller than the line, forcing some text to drop below. This requires more page turns (due to the inefficiency at displaying text) and is more challenging to read.

The vast majority of fanfics there are completely unreadable when saved, even on the computer, as the line breaks are replaced with "undisplayable" characters (i.e. those little boxes like when you have Chinese characters and don't have the font sets installed). Text wraps continously with no line breaks whatsoever, and every paragraph is only separated by two boxes, with more boxes in the middle of sentences where the original forced line breaks were.

Now, I understand why this is the way it is; when read online at the Archive, the fics are very readable and exactly as designed. However, since I'd like to read some fics offline, I'm going to have to go through them and edit all those line breaks out so there are just paragraph separations. While I'm at it, I might as well make them epub format, which is basically xhtml in a zip container; it's very small file size. I took a children's book and converted it just the other day, and the epub size was less than half of the txt size (122 kb to the original 267 kb). Epub has another benefit in that it lets you create a TOC and jump to individual chapters.

Since I'm going to do this for various fanfics, is there any way the Archive would want to host epub versions of fanfics as well? Should I post them somewhere online? Since this is simply re-formatting (with some chapter breaks--which I can use as they were posted on the mbs or just find scene breaks every so many pages, roughly), I don't see as big an issue with it as with audio fanfics, or translations, but would there be permission issues? Do I need to just keep it to myself, or is it something I could share with other FoLCs who also have e-book readers and would like nicely formatted epubs they can read offline?

I would welcome any thoughts/comments any of you have on this, particularly staff input as much of this is a question only they can truly answer. (I'm also curious how many of you read fanfics on an e-book reader and would find this sort of thing useful. Anyone else already doing this type of thing?)

Hi,

Have you tried using Stanza to change the stories into ePub format? I have used Stanza's desktop program for the conversion and then read the stories using the Stanza app on my iPod Touch. I have found the stories so converted to be extremely readable. I have never had to tweak line endings at all. (The current version of Stanza doesn't even require translation into the ePub format; but I had used that format with an earlier version of Stanza, back when its app it enable file transfers over a wired LAN via a separate third party program.)

Joy,
Lynn

I use Good Reader on my iPod Touch, works really well. Can read stories in many formats, including PDF, plain TXT, Word documents, etc.

I briefly considered trying Stanza, but I don't have an iPod Touch or anything of the like, and I couldn't figure out where to download it. I'm also not fond of anything too Mac-centered (I actually figured out how to install just the necessary files for my iPod Nano to communicate with my computer without installing iTunes itself, which is a memory hog on Windows and not a program I want at ALL--I'm using foobar2000 to manage my iPod library instead).

I poked around, and ended up going with Sigil, which had such a shallow learning curve that I've got one children's book converted and am working on a second, in about two days' time, TOC and all. I haven't tested it on the fanfics yet, but I plan on editing those line breaks in the txt file anyhow so I don't have them hanging around all uglified. If I try it in Notepad++ or some other Notepad replacement, I hope to be able to do some better search&replace, since Notepad won't let me replace boxes with carriage returns.

The Astak EZ Reader can read a whole lot of formats too, actually--that was a big reason why I wanted to get it over other e-book readers (price was the other one)--but not all of them have all the features. Some you can't search in, and a lot of them don't have a TOC or anything like that. PDFs are close in features to epubs (though some look atrocious due to the nature of pdfs), but epub is better than the rest because it has search, TOC, and more zoom levels altogether. Txt format is very limited on the Astak EZ Reader, so I would need to convert to something else if I wanted to take advantage of any of the possibilities (TOC would be nice with some of the gigantic novels--I can't imagine trying to read Masques, for instance, without some sort of chapter system--what if I wanted to just read from a certain point on?). Epub is a natural choice because it's the best-supported format on the Astak, and Sigil is an easy program to work with. I can make sure it will look good and be formatted nicely when I'm done.

The desktop version of Stanza quickly and easily converts documents; you don't need an iPod to use it. It's available at http://www.lexcycle.com/desktop . To convert a document, all you need do is open the plaintext version of the document go to File, choose "Save As," and select the name and location you wish the ePub version to be placed. Couldn't be easier!

But it sounds like you have some other ideas in mind. Whatever you go with, I hope you do find something that will work for you.

Joy,
Lynn

This may help you, Doranwen. I've got a Sony Reader and it uses "PDF file, EPUB file, Text file, RTF file, Word file (unsecured)*1,
BBeB file".
Also I'm on a PC with System 7 right now.
For the Archive files I convert them to Word to read on my computer and then convert them to RTF to read on the Reader. There are no line break problems or format issues there. It works just fine. I'm reading "Aussie Rules" on it now. True, that's from these MBS, not the Archive. But the same thing works.
It's simple, it's easy and doesn't require anyone's permission. Give it a go. File size doesn't seem to be an issue with my reader.
regards
Artemis

I think there's a bit of confusion here about what I'm asking.

The Astak EZ Reader supports over a dozen formats, including TXT, RTF, PDF, DOC (though VERY poorly! I won't use it because it treats font sizes as pixel sizes, necessitating a major font size change document wide), and, of course, EPUB. I don't need to change the file format to be able to read it in the first place.

However, I would like to have TOC with the larger stories (I gave Masques as an example of a size that would be very handy to have some sort of chapter system to jump to if I wanted). Txt might support search (haven't tested it), but not TOC. It also has an ugly-looking font compared to the epub standard font, so I'd rather not keep files as txt. Rtf is not the best format to use either, since it lacks TOC as well, I believe. As epub is best supported by the Astak, I am choosing to convert to epub. I will probably do this for my own personal enjoyment whether anyone else does so, or whether I can share it or not. File size is not a concern on my end as I have a very large SD card that the reader accepts, meaning I can store my entire library and then some, when it comes to stories to read.

I do have some issues with converting line breaks, and those will be issues whether I copy to Word or just edit in the txt file. Even copying a fic straight from the Archive to Word does NOT eliminate the forced line breaks, meaning I have a lot of work to do to make it flow across each line naturally until the paragraph ends. Most saved txt files replace those line breaks with strange boxes, and I can't imagine any program fixing those automatically. If Stanza does, that IS interesting, but it doesn't appear it is a program I can use. Doing a search a little while ago seemed to imply that one had to get it through iTunes, and I don't plan on installing iTunes on my computer for any reason. The best solution would be to work on some sort of macro that would do the editing for me. I'm not so good at setting those up, but with the amount of fics I'd like to have at my fingertips in epub format, I think it would be worth it for me to learn. That might be the subject of a separate post in Off Topic.

Anyway, since I plan on creating epubs for my personal reading anyhow, the real purpose of this post was to ask if the epubs I create this way could be shared somehow, whether anyone would WANT a copy of them, whether it would affect permissions, etc. I mentioned the very small file size because if someone were to host them, small files are always preferred over larger ones. Knowing they take up less room than a txt file (which is, admittedly, smaller than most since it only encodes a byte per character, with no overhead unlike Word's 31 kb overhead) is helpful in that respect.

It is interesting to hear that the fics seem to be very readable to the rest of you as-is. I haven't figured out how that works, unless the forced line breaks are just right so they barely are noticeable. In my case, the screen I read on is somewhere about 2/3 (I think?) of the line, requiring a lot of alternating full lines and short lines of just a couple words. This is distracting and difficult to read, plus it stretches the file out to many more pages, requiring much more frequent page-turning (and I'm already a very quick reader!). This is not ideal, and worth the effort to fix the files, in my opinion.

Thanks for the input so far!

To get rid of the undesired line breaks:

Cut and paste into word.

Search and replace - ^p^p with @@@ (or some other random thing that won't naturally occur in the doc).

Search and replace - ^p with either nothing or a single space (do one with no space before the replace all and see if where it replaced it you have twowords with no space - if so, you need a space; if not, no space necessary).

Search and replace - @@@ with ^p^p

Should fix all the bizarre ones without undoing the extra hard return between paragraphs.

No idea how to help you on TOC, though. Have you tried Calibre? It...*looks* like there's a TOC option, but I don't know how to work it.

Good luck!

Bethy

Hi Dorenwen:
Yes, I did understand what you were asking.

regards
Artemis

Artemis,

Ereaders and Word will do the word-wrap correctly in an .rtf file if the line is too long. If I understand correctly, Doranwen's problem is where there are extra hard returns, causing some lines to be too short.

For example the paragraph would look
like this
and then keep going for a full line
and then
another short one, etc and so forth.

(With the original having been:
For example the paragraph would look like this
and then keep going for a full line and then
another short one, etc and so forth.)

It's the extra hard return after *this* and *then* that are causing the problems.

At least, I think that's what she means - Doranwen? Am I right?

Bethy

Yes, Bethy, I did understand that. I did various experiments and found it didn't bother me enough to go to any effort to deal with it.
Whether the Archive can be put into epub is up to LabRat and crew.

Artemis

Aha! I didn't know that ^p would represent the carriage return (though now that I think about it, I should have known there would be *something*). YES! Now I can fix them. Thank you SO much, Bethy! (The fics do need that space replacement, I found, so I'll remember that for the future.)

I do have Calibre--I use it for quick conversions when I just need something readable and searchable and I don't care about it having a great TOC. It will create a TOC automatically if the file is formatted just right--I'm not quite sure how to do that, though, and it's a bit touchy about it.

As for the TOC, Sigil creates them very easily. The only thing I have to do is mark headings on the text. It'll even do multi-layer depth to a TOC. For instance, I could have a Part I and Part II, and under each of those have separate chapters, so would be able to choose which part, then which chapter. I also like the polished and finished aspect of epubs better.

And yes, Bethy, that's EXACTLY the problem--you illustrated it perfectly. My screen is small enough that I had major issues with it. I suspect that my fast reading pace exacerbated it beyond what would frustrate most people. I timed myself on Long Strange Trip once, actually, and it took me only about 4 1/2 hours to read the whole thing. At that sort of pace, the constant flicking of the page turn is not only annoying, it gets tiring on the fingers AND drains the battery quickly. Fixing the carriage return problem will do a lot to help. (And converting to epub will give me a nice TOC and a better viewing font. )

Having all the technical ability of a drunk wombat, you lost me after e-book, Doranwen. .

But I will pass this along to Lauren - who has all the technical ability of Rodney McKay - and see what she thinks.

LabRat (quite pleased with herself to have managed to work her current Atlantis obssession into this post....)

Glad it was helpful, Doranwen!

The carriage return is something that bothered me even on regular computer screen reading - in that every line was too short. I can't take full credit for the fix though - someone else on the boards, at some point a long time ago, explained how that works.

So I was already doing it and then when I got a nook, I kept doing it, because yes, the problem is exacerbated on the smaller screen.

Also, just as a note in case you get fic from other websites - some don't have a hard return but a... "soft return"? I think that's what it's called. In Word, you create it by holding down shift as you hit enter. If you turn on the paragraph/space/etc indicators in Word (a little paragraph sign P on the menu), it'll show up as an arrow that turns at a right angle (like the one on your enter key).

IF you get that, you need to replace ^l (lowercase L) with ^p. Yes, it took me hours and multiple friends to figure that out. I nearly cried before we got to the solution, because I had just pulled some long fic that I really, really wanted to read on my nook and it was full of the ^l returns.

(ETA - the ones I was having a problem with were not mid-paragraph, but the normal carriage returns at the end of paragraphs. The problem was, the nook ignored them altogether and so I ended up having one solid massive humongous paragraph. Just...in case that shows up differently/makes a difference to you.)

And Labby - lol at the working in your current obsession. Nice job!

Bethy

When I saw this thread, my thought was. "Exactly! I would love to have the stories available in epub format."

But then as I read the thread the replies all seem to be about how to convert txt files to epub. I have done that with... I don't know, somewhere about 50 to 100 stories. I convert them for my ebook reader.

So, let me try a more direct question:

Would it be possible to start an effort to convert existing stories to epub and set up an epub section on the archive?

This would be ideal for me. I'd be happy to work on the project and donate the ones I've converted. At the very least I have all of my own stories in epub.

So, I'd be curious to hear some reaction to this.

Bob

Re the issue of getting a TOC for a story, for Word 2007 and later there is a "Document Map" on the left side of the window that is a TOC to any level you want.
Bob, since you've converted so many txt files to epub, I think folcs on this thread would be interested in how exactly you did that.
regards
Artemis

Re: I don't have any magic. I use the Stanza desktop application on my Windows PC.

If I want a quick-and-dirty conversion, I load the txt file and do a Save As to save in epub foramt exactly as Lynn describes above.


If I want a clean conversion, I do this:

1. Load the txt file into MS Word

2. Use a technique almost exactly as Bethy describes to remove the hard-returns. Any yes, you need to be ready to handle the soft-return variant. It's just a different search but once you find the character it works perfectly every time. (And my experience in figuring this out was very similar to Bethy's.)

3. Then I have one more trick. For each section, add the word "Chapter x" to open each section. Then save the whole thing as MS Word format. (The is partly why I started using the word "Chapter" in my longer stories. It saves ME a step )

4. Then load the .doc into the Stanza desktop app. About 2/3 of the time Stanza will see the "Chapter" key words and create the chapter breakpoints. I don't know why it doesn't work all the time. I haven't tracked that down yet. If someone knows the magic for this part, I'd love to know.

Then Save As the epub. If Stanza recognized the chapter breaks, you will have a properly formatted, chapterized epub version.

Yes, this takes a little time but once you've done it a few times, it's pretty quick. I'd say less than 10 minutes for most files. The hard part is breaking into chapters if there isn't a good substitution word.

Bob

I've used Stanza on my mac but haven't tried to add a toc to anything. However since you say it works most of the time but not all it is probably something like needing 2 newines in a row and starting in column 1, and maybe having a newline right after the number. Two newlines in a row is a common this starts a new paragraph marker.

You might see what is a good Windows text editor (not word processor) and look at the file in a mode where it shows every character even the "invisible" ones like newline, tab etc. Than you can see what the difference is between ones that work and ones that don't.

Yep. I tried all that and more. I figured it had to be something, but you reach a point where you just have to shrug and admit that it isn't worth it. I may take another look at the file with a binary hex editor and see if I can crack the code, but I haven't had time yet.

Edit: Sorry if the above sounds flip. I just got frustrated that I couldn't find the pattern. I'm pretty sure that I looked at all these variants and others. I usually figure these things out in short order. I don't like the fact that I don't have the answer.

Bob

Oh, Sigil is so easy to use that I wouldn't want to do it in an auto-conversion format (Calibre has turned out very questionable contents--very little TOC recognition overall, too). Sounds like Stanza doesn't always recognize the Chapter markers, whereas I can name a chapter anything I feel like (granted, in fanfic, it'll always be Chapter 1, Chapter 2, etc. unless the author had fancy names for each part), and just apply a heading format (H1, H2, H3, etc.--H1 is usually for titles, H2 for author, H3 for chapters, and H4-H6 for subheadings if needed). Sigil automatically recognizes those, and I can go into a TOC editor (a simple dialog box showing the list of chapters) and uncheck any I don't want as sections to jump to.

In the last three days since I started using it, I've created epubs out of two children's books (they required extensive editing to fix scanning errors, basically entailing re-reading them), and today, when I didn't need to do major editing beyond replacements, I created epubs out of one book, two poems, and a short story, in between all the cooking I've been doing. That one book only required a simple scan for each chapter marker, insert chapter break, highlight title and mark heading, and go to the next chapter to get a TOC formatted nicely. Very little work and produces an excellent result. So I could create tons of epubs that way, if the Archive would want to host them. I'll probably create a good lot *anyway* since I want to be able to re-read all my favorites off-computer (and I have a LOT of favorites!).

And aside from fixing the line break issue (which I can't thank Bethy enough for), my real question was indeed whether an epub option on the Archive (kind of like the audiofic option, perhaps?) is something that might be done. It would appear that there are a few of us willing to create these epubs and format them nicely for the Archive, which only leaves the question of whether the Archive would want to store them. Obviously there would need to be some new coding to fit them in, so that creates more work for people . . . I think it would be worth it, though, since more and more of us have e-book readers which read epubs, and computer screens are so bad for the eyes if you want to sit there for hours glued to those addicting epics. (Forget printing them out for real; I would still love to be able to hold some of them as books in my hands, but until I get equipment to do my own print-on-demand stuff, which will probably happen the day pigs learn to fly since that takes a lot of money I don't have, that isn't a realistic dream.)