Commit Graph

59 Commits

Author SHA1 Message Date
Chocobozzz
62549e6c98
Rewrite youtube-dl import
Use python3 binary
Allows to use a custom youtube-dl release URL
Allows to use yt-dlp (youtube-dl fork)
Remove proxy config from configuration to use HTTP_PROXY and HTTPS_PROXY
env variables
2021-10-22 10:25:24 +02:00
Chocobozzz
02b286f890
More robust webtorrent redundancy download
Avoid issues with inconsistencies between magnet infohash and torrent
infohash, blocking webtorrent upload that will timeout
2021-09-06 16:19:49 +02:00
Chocobozzz
41fb13c330
esModuleInterop to true 2021-08-27 15:12:22 +02:00
Jelle Besseling
0305db28c9
Add support for saving video files to object storage (#4290)
* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests

Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-08-17 08:26:20 +02:00
Chocobozzz
679c12e69c
Improve target bitrate calculation 2021-08-06 14:13:26 +02:00
Chocobozzz
d26836cd95
Refactor notifier 2021-08-02 10:39:51 +02:00
Chocobozzz
83903cb65d Generate random uuid for video files 2021-07-26 11:29:31 +02:00
Chocobozzz
765b150245
Fix lint 2021-06-08 10:19:04 +02:00
Chocobozzz
ea54cd04c1
Fix video upload with a capitalized ext 2021-06-08 10:01:50 +02:00
Chocobozzz
2539932e16
Instance homepage support (#4007)
* Prepare homepage parsers

* Add ability to update instance hompage

* Add ability to set homepage as landing page

* Add homepage preview in admin

* Dynamically update left menu for homepage

* Inject home content in homepage

* Add videos list and channel miniature custom markup

* Remove unused elements in markup service
2021-05-27 15:59:55 +02:00
Chocobozzz
1bcb03a100
Use a class for youtube-dl 2021-05-11 11:32:31 +02:00
Chocobozzz
8efc27bf14 Cleanup 2021-02-18 13:38:09 +01:00
Chocobozzz
90a8bd305d Dissociate video file names and video uuid 2021-02-18 13:38:09 +01:00
Chocobozzz
a35a22797c Remove previous thumbnail if needed 2021-02-16 10:36:44 +01:00
Chocobozzz
e3b4c084cd
Guess if we need to generate the thumbnail for imports 2021-02-12 09:37:01 +01:00
Chocobozzz
44d1f7f2e8
Painfully debug concurrent import jobs 2021-02-09 11:46:30 +01:00
Chocobozzz
77d7e851dc Add priority to transcoding jobs
(1 = highest priority)

100 for new resolutions
10 for original file optimization

Add a malus for transcoding jobs depending on how many uploads the user did in the
last 7 days
2021-01-25 14:38:52 +01:00
Chocobozzz
805b8619c1
Fix (again) youtube import 2021-01-19 16:36:21 +01:00
Rigel Kent
454c20fa7c
stricter youtubedl format selectors (#3516)
* stricter youtubedl format selectors

make sure selectors avoid av1, and otherwise match as closely to the
maximum resolution enabled for transcoding

* add support for merge formats in youtubedl

* avoid vp9.2 in youtubedl to avoid any HDR

* move getEnabledResolutions, safer replace of imported extension

* add test for youtube-dl selectors
2021-01-15 15:56:56 +01:00
Chocobozzz
daf6e48010 Split ffmpeg utils with ffprobe utils 2020-11-25 10:07:51 +01:00
Chocobozzz
fb7194043d Check live duration and size 2020-11-09 15:33:04 +01:00
Chocobozzz
26d6bf6533
Split types and typings 2020-06-18 10:46:27 +02:00
Rigel Kent
2158ac9034
Add server plugin filter hooks for import with torrent and url (#2621)
* Add server plugin filter hooks for import with torrent and url

* WIP: pre and post-import filter hooks

* Rebased

* Cleanup filters to accept imports

Co-authored-by: Chocobozzz <me@florianbigard.com>
2020-05-14 11:10:26 +02:00
Chocobozzz
8dc8a34ee8 Avoir some circular dependencies 2020-05-04 16:21:39 +02:00
Kim
b1770a0af4
Add thumbnail / preview generation from url on the fly (#2646)
* Add thumbnails generation on the fly to URL import

* Display generated preview to import first edit

* Use ternary to get type inference

* Move preview/thumbnail test just after import

Co-authored-by: kimsible <kimsible@users.noreply.github.com>
2020-04-20 10:28:38 +02:00
Chocobozzz
d57d1d83c6
Support audio files import 2020-04-03 15:41:39 +02:00
Chocobozzz
a15871560f
Move to eslint 2020-02-03 08:31:02 +01:00
Chocobozzz
d7a25329f9
Add ability to disable webtorrent
In favour of HLS
2019-11-25 10:59:43 +01:00
Chocobozzz
8424c4026a Add auto follow back support for instances 2019-09-04 16:24:58 +02:00
Chocobozzz
0283eaac2a
Cleanup model types 2019-08-20 14:53:42 +02:00
Chocobozzz
453e83ea5d
Stronger model typings 2019-08-19 17:26:35 +02:00
Chocobozzz
5b77537ce5 Correctly notify on auto blacklist 2019-07-24 10:58:16 +02:00
Chocobozzz
536598cfaf
Add audio support in upload 2019-05-16 17:16:47 +02:00
Chocobozzz
3acc508440
Upgrade sequelize 2019-04-24 16:26:21 +02:00
Chocobozzz
e8bafea35b
Create a dedicated table to track video thumbnails 2019-04-24 16:25:52 +02:00
Chocobozzz
74dc3bca2b
Don't expose constants directly in initializers/ 2019-04-11 14:26:41 +02:00
Chocobozzz
6dd9de95df
Move config in its own file 2019-04-11 13:45:39 +02:00
Josh Morel
7ccddd7b52 add quarantine videos feature (#1637)
* add quarantine videos feature

* increase Notification settings test timeout

to 20000ms. was completing 7000 locally but timing out
after 10000 on travis

* fix quarantine video test issues

-propagate misspelling
-remove skip from server/tests/client.ts

* WIP use blacklist for moderator video approval

instead of video.quarantine boolean

* finish auto-blacklist feature
2019-04-02 11:26:47 +02:00
Chocobozzz
a0327eedb0
Rename video-file job to video-transcoding 2019-03-19 17:00:08 +01:00
Chocobozzz
dc13348070 Add import finished and video published notifs 2019-01-09 11:15:15 +01:00
Chocobozzz
e8d246d526 Add notification settings migration 2019-01-09 11:15:15 +01:00
Chocobozzz
cef534ed53 Add user notification base code 2019-01-09 11:15:15 +01:00
Chocobozzz
f481c4f9f3
Use move instead rename
To avoid EXDEV errors
2018-12-11 15:12:38 +01:00
Chocobozzz
6040f87d14
Add tmp and redundancy directories 2018-12-04 16:04:15 +01:00
Chocobozzz
58d515e32f
Fix images size when downloading them 2018-11-16 16:48:17 +01:00
Chocobozzz
cf9166cf2f
Add timeout on youtube dl to cleaup files 2018-10-01 12:00:05 +02:00
Chocobozzz
e95e0463d8
Correctly delete files on failed import 2018-10-01 10:44:16 +02:00
Chocobozzz
627621c1e8
Optimize SQL requests of watch page API endpoints 2018-09-19 09:54:37 +02:00
Chocobozzz
62689b942b
Correctly migrate to fs-extra 2018-08-27 16:23:34 +02:00
Chocobozzz
541006e355 Correct webtorrent download cleanup 2018-08-08 09:30:31 +02:00