This is a limitation of Github itself. If you have large files like the one you have, you need to store them outside of the git tree (or use Git LFS). You could store your EPUB and DOCX document in another url and provide a link to download in your HTML book.
You could also try reduce the size of all images by optimizing them in case they are too big and the source of the size to the files.
You could also try another hosting servive too with more space.
Hope it helps