1
00:00:05,190 --> 00:00:05,990
Hello.

2
00:00:07,090 --> 00:00:11,450
So I'm going to talk about download management
on metered connections, which

3
00:00:11,470 --> 00:00:16,090
is something that I was looking at recently at
Endless as part of our work

4
00:00:16,110 --> 00:00:19,277
on Endless OS. So what is the problem?

5
00:00:19,537 --> 00:00:22,957
Many people, especially in sort of non-Western
countries use metered internet

6
00:00:22,977 --> 00:00:25,237
connections where they get charged for the
bandwidth they use.

7
00:00:25,677 --> 00:00:29,637
This means that they can't download everything
they want to, when they want

8
00:00:29,657 --> 00:00:33,657
to, and often they have to be quite careful
and in control of what they download

9
00:00:33,677 --> 00:00:37,217
each month. They have to mentally prioritise
certain things over other things,

10
00:00:37,237 --> 00:00:40,107
stop some things from being downloaded,
especially if some of those updates

11
00:00:40,127 --> 00:00:44,507
are quite large, like operating system updates
or Flatpak updates or whatever.

12
00:00:44,827 --> 00:00:49,447
As part of our target market, Endless has been
quite interested in making

13
00:00:49,467 --> 00:00:52,827
it easier for people with such internet
connections to download what they

14
00:00:52,847 --> 00:00:55,967
want and when they want and without causing
them huge bandwidth bills, or

15
00:00:55,987 --> 00:01:00,247
at least making them aware and in control of
those bandwidth bills.

16
00:01:00,917 --> 00:01:03,757
It's worth noting that what I'm discussing
during this talk only applies

17
00:01:03,777 --> 00:01:06,717
to quotation marks, big downloads.

18
00:01:07,037 --> 00:01:11,367
The definition of big is not really defined,
but think of it what you will,

19
00:01:11,387 --> 00:01:13,227
like OS updates, Flatpak updates.

20
00:01:15,057 --> 00:01:19,337
The solution that we've come up with is a sort
of voluntary service for software

21
00:01:19,357 --> 00:01:21,557
components to use. It's not like a network
firewall.

22
00:01:21,577 --> 00:01:25,117
It's not enforced queuing. It's not a
mandatory thing that applies to everything.

23
00:01:25,317 --> 00:01:30,397
It's a kind of opt-in service that certain
applications can use to query

24
00:01:30,417 --> 00:01:32,977
whether they should do a download and when.

25
00:01:33,717 --> 00:01:37,697
So what is metering? I'm going to give a bit
more detail on the definition

26
00:01:37,717 --> 00:01:40,297
of metered connections as I see it.

27
00:01:40,317 --> 00:01:43,937
They're connections where you pay per unit of
bandwidth that you use rather

28
00:01:43,957 --> 00:01:46,997
than paying a fixed price per month for
unlimited bandwidth.

29
00:01:47,427 --> 00:01:51,387
Pricing structures vary a lot. Some tariffs
allow unlimited downloads at

30
00:01:51,407 --> 00:01:55,287
a certain time of day, for example early in
the morning, overnight for OS

31
00:01:55,307 --> 00:01:59,887
updates. Or they might allow downloads from
certain websites for free and

32
00:01:59,907 --> 00:02:02,887
not from others, which is not net neutral, but
it happens.

33
00:02:03,207 --> 00:02:07,687
And each network provider, and there might be
many network providers, might

34
00:02:07,707 --> 00:02:10,707
have several tariffs. Those tariffs might
change regularly.

35
00:02:10,727 --> 00:02:15,087
They might apply promotions during certain
months for certain non-net neutral

36
00:02:15,107 --> 00:02:17,447
websites or whatever. There's all sorts.

37
00:02:17,767 --> 00:02:23,247
Specific examples would be a tariff that
allows free access to Facebook but

38
00:02:23,267 --> 00:02:24,037
not to other websites.

39
00:02:24,577 --> 00:02:28,787
Does it affect me? Most people in this room,
you probably wouldn't immediately

40
00:02:28,807 --> 00:02:29,937
think that it does affect you.

41
00:02:31,067 --> 00:02:34,267
But it kind of does in certain ways.

42
00:02:34,647 --> 00:02:37,987
It's typical to have an unlimited home or work
internet connection for most

43
00:02:38,007 --> 00:02:38,967
of us I think,

44
00:02:39,567 --> 00:02:42,807
but you can encounter metered connections on
trains, certainly in the UK,

45
00:02:43,007 --> 00:02:45,667
where you sort of get on the train and unless
you have a first-class ticket

46
00:02:45,687 --> 00:02:47,167
you have to pay to use the internet.

47
00:02:48,327 --> 00:02:52,347
Or public Wi-Fi hotspots in airports, you can
pay for internet access if

48
00:02:52,367 --> 00:02:54,727
they haven't given you free Wi-Fi.

49
00:02:54,907 --> 00:02:58,807
Many mobile internet tariffs, unless you've
got a contract, will make you

50
00:02:58,827 --> 00:03:04,047
pay per gigabyte. So it does kind of affect
the more powerful Linux desktop

51
00:03:04,067 --> 00:03:05,467
user in some ways.

52
00:03:06,087 --> 00:03:08,887
But there are other ways that a connection can
benefit from prioritising

53
00:03:08,907 --> 00:03:11,907
downloads. For example, if you're on a home
internet connection that's shared

54
00:03:11,927 --> 00:03:17,467
between the whole street in terms of its
uplink to the internet provider,

55
00:03:17,557 --> 00:03:20,557
the number of parallel downloads you have, you
might want to limit it so

56
00:03:20,577 --> 00:03:23,137
that some of them complete first. Certainly,
the important one's complete,

57
00:03:23,157 --> 00:03:26,057
you might want to prioritise your OS updates
below everything else.

58
00:03:26,077 --> 00:03:28,877
You might want to prioritise the film that
you're downloading or whatever

59
00:03:28,897 --> 00:03:31,397
before that, or you're streaming TV.

60
00:03:31,637 --> 00:03:35,467
Who knows? You also might want certain
downloads not to start until you're

61
00:03:35,487 --> 00:03:36,247
back home.

62
00:03:36,767 --> 00:03:39,687
Like when you're traveling, you don't really
care about certain things being

63
00:03:39,707 --> 00:03:42,447
downloaded. You can delay them until later.

64
00:03:42,712 --> 00:03:47,252
So, we wrote Mogwai, it's a download
scheduler, applications which want to

65
00:03:47,272 --> 00:03:51,692
do big downloads, tell the scheduler what they
want to download, so it's

66
00:03:51,712 --> 00:03:54,772
size, the domain name you're downloading it
from, or quite a metadata like

67
00:03:54,792 --> 00:03:58,882
that. The scheduler, it schedules all the
incoming requests for downloads,

68
00:03:59,382 --> 00:04:02,082
and it signals each application when it's time
to start downloading.

69
00:04:02,502 --> 00:04:06,282
Scheduler can preempt downloads, tell an
application to pause its download

70
00:04:06,302 --> 00:04:11,612
and carry on later. And then you can get
prioritisation in more real time.

71
00:04:11,902 --> 00:04:14,072
The scheduling decisions are based on various
things.

72
00:04:14,602 --> 00:04:17,802
Amongst those, there's a tariff file for each
network connection that says

73
00:04:17,822 --> 00:04:21,442
what period of day the connection starts being
metered, stops being metered,

74
00:04:21,462 --> 00:04:24,662
whether it's metered all the time, unmetered
all the time, whether there

75
00:04:24,682 --> 00:04:29,302
are any bandwidth limits or capacity limits
during those periods, and the

76
00:04:29,322 --> 00:04:32,442
format is extensible so that other properties
could be added in future to

77
00:04:32,462 --> 00:04:35,072
describe more and more complicated network
tariffs as

78
00:04:35,782 --> 00:04:39,492
we encounter them. For example, we don't
currently implement it, but we could

79
00:04:39,512 --> 00:04:40,182
start supporting

80
00:04:41,252 --> 00:04:45,143
downloads on certain domain names being
metered or unmetered and that kind

81
00:04:45,163 --> 00:04:49,506
of non-net neutral behavior. As I said before,
it is not security.

82
00:04:49,526 --> 00:04:55,446
It's an opt-in system, it's a D-Bus service
that things can request download

83
00:04:55,466 --> 00:04:58,046
scheduling from and they're free to ignore the
answer.

84
00:04:58,306 --> 00:05:01,626
There's no enforcement going on. And it
doesn't currently monitor bandwidth

85
00:05:01,646 --> 00:05:02,526
usage either.

86
00:05:03,066 --> 00:05:06,086
So it's not a closed feedback loop.

87
00:05:06,796 --> 00:05:10,946
It takes in the sizes that the applications
say, I'm gonna download one gigabyte,

88
00:05:10,966 --> 00:05:13,426
I'm gonna download two gigabytes. It sums them
all together.

89
00:05:13,446 --> 00:05:18,206
It calculates that against the tariff limit,
and then it assumes that the

90
00:05:18,226 --> 00:05:21,566
applications play fair and that the sizes
don't change, or that if they do,

91
00:05:21,586 --> 00:05:23,066
the applications notify of it.

92
00:05:23,631 --> 00:05:26,181
But we could do work in future to improve
that.

93
00:05:26,411 --> 00:05:30,031
How I use it? So as I said, it's a D-Bus
service.

94
00:05:30,051 --> 00:05:34,181
It sits on the system bus. Any application can
request things from it.

95
00:05:34,321 --> 00:05:38,581
It's currently used in Endless OS in our fork
of GNOME software and also

96
00:05:38,601 --> 00:05:41,351
in our OS updater, which is called EOS
updater.

97
00:05:41,451 --> 00:05:44,621
We have changes to GNOME Control Center to add
it to the automatic updates

98
00:05:44,641 --> 00:05:49,401
panel, but none of this is sort of exclusive
to us.

99
00:05:49,421 --> 00:05:54,181
If you want to use it, you can use the API
from the libmogwai-schedule-client,

100
00:05:54,201 --> 00:05:56,341
first bullet point, library.

101
00:05:56,941 --> 00:06:00,961
And that's a client for the D-Bus API, so
that's sort of standard D-Bus mechanics.

102
00:06:01,201 --> 00:06:05,421
There's an example for how to do that in the
mogwai-schedule-client utility,

103
00:06:05,441 --> 00:06:09,081
which is a command line utility that's
essentially curl, says,

104
00:06:09,761 --> 00:06:14,841
run this, download this file, save it here,
but it does it in a scheduling-aware

105
00:06:14,861 --> 00:06:18,741
manner. So if the daemon says, no, don't
download that now, it won't download,

106
00:06:18,761 --> 00:06:20,331
it'll block until the daemon says yes.

107
00:06:20,731 --> 00:06:24,081
But there's example code in there which you
can start basing things off.

108
00:06:24,411 --> 00:06:28,491
The PDF of these slides will contain links to
that code afterwards, so if

109
00:06:28,511 --> 00:06:31,761
you wanna download it later and check out
those links, you can.

110
00:06:31,811 --> 00:06:34,871
But essentially, your application needs to
create a schedule entry for every

111
00:06:34,891 --> 00:06:37,011
download that you care about doing.

112
00:06:37,521 --> 00:06:40,781
Set the metadata of the download, like the
size, the domain, whatever.

113
00:06:41,461 --> 00:06:46,621
And then you need to tie the state of your
download to signals from the scheduler,

114
00:06:46,641 --> 00:06:49,801
telling it when to start the download, when to
pause, when to resume, and

115
00:06:49,821 --> 00:06:52,431
then you need to update the scheduler saying,
yep, I've finished now, or

116
00:06:52,451 --> 00:06:55,021
actually, I want to cancel this download,
forget about it.

117
00:06:55,041 --> 00:06:59,721
Here is a screenshot of what we have in
Endless OS for scheduling downloads

118
00:06:59,741 --> 00:07:02,501
or scheduling OS updates in the Control
Center.

119
00:07:03,461 --> 00:07:08,481
So, you can see in this case, I've got an
unlimited connection, no metering,

120
00:07:09,071 --> 00:07:12,331
but I can still schedule my updates to only
happen during certain times of

121
00:07:12,351 --> 00:07:15,931
day. If I were to connect to a metered
connection,

122
00:07:16,631 --> 00:07:19,571
then the tick box in the middle would start to
take effect

123
00:07:20,151 --> 00:07:22,721
and the scheduling would still continue to
take effect.

124
00:07:22,872 --> 00:07:25,442
What is left to do? It's quite a big scope
problem.

125
00:07:25,742 --> 00:07:28,422
There's lots of things you can add to it, lots
of tweaks you can make, lots

126
00:07:28,442 --> 00:07:31,462
of improvements to the scheduling you can do,
lots of other sources of data

127
00:07:31,482 --> 00:07:35,832
you can pull in. The next big task on the list
is to add a monitoring daemon,

128
00:07:36,152 --> 00:07:42,362
which would sit sort of near to the systemd
level of the stack and use, what

129
00:07:42,382 --> 00:07:48,472
Per-cgroup monitoring of the network
connection usage to build a log

130
00:07:48,492 --> 00:07:53,532
of how much is being downloaded per
application and store that somewhere.

131
00:07:53,552 --> 00:07:57,732
That information can then be used by the
scheduler to close the feedback

132
00:07:57,752 --> 00:08:01,332
loop and say, yep, you've reached your limit
for this download period, everything

133
00:08:01,352 --> 00:08:06,322
else gets cancelled now. But it could also be
used by the GNOME usage application,

134
00:08:06,342 --> 00:08:10,022
for example, to show nice pretty graphs of
what's being downloaded by each

135
00:08:10,042 --> 00:08:14,982
application, how near to each limit each
application is.

136
00:08:15,012 --> 00:08:17,002
I've got a plan for how this can be
implemented efficiently.

137
00:08:17,022 --> 00:08:21,658
Currently, what GNOME usage does is not
particularly efficient.

138
00:08:21,738 --> 00:08:26,438
The plan is using nftables and systemd unit
hierarchy, and it should be fairly

139
00:08:26,458 --> 00:08:28,678
straightforward to do, but I haven't had time
to do it yet.

140
00:08:28,788 --> 00:08:31,948
Aside from that, the scheduler can always be
made more clever.

141
00:08:31,968 --> 00:08:36,108
It can take more factors into account, more
different tweaks for when to

142
00:08:36,128 --> 00:08:38,418
start, when to stop, what to prioritise above
what else.

143
00:08:38,748 --> 00:08:41,728
It's a fairly open-ended problem. And I'm
willing to mentor people to implement

144
00:08:41,748 --> 00:08:45,028
this functionality in Mogwai because my time
on it has been significantly

145
00:08:45,048 --> 00:08:48,668
reduced. But mentoring is always something I'm
happy to do.

146
00:08:48,735 --> 00:08:51,265
And that is that. Any questions?
