{"id":127,"date":"2023-10-12T16:48:46","date_gmt":"2023-10-12T20:48:46","guid":{"rendered":"https:\/\/www.aarondefazio.com\/tangentially\/?p=127"},"modified":"2023-10-18T11:26:17","modified_gmt":"2023-10-18T15:26:17","slug":"d-adaptatation-wins-an-outstanding-paper-award-at-icml-2023","status":"publish","type":"post","link":"https:\/\/www.aarondefazio.com\/tangentially\/?p=127","title":{"rendered":"D-Adaptatation Wins an Outstanding Paper Award at ICML 2023!"},"content":{"rendered":"

Konstantin and I were honored with an ICML Outstanding paper award for our
\npaper \u201cLearning-Rate-Free Learning by D-Adaptation\u201d. Besides the award, our
\nimplementation has already been downloaded nearly a million times over the last
\nyear (most before the award), and is seeing wide-spread use in training and
\nfine-tuning deep learning models, particularly LoRAs.\n<\/p>\n

<\/p>\n

Read on if you would like to know more about our work and the impact it is
\nalready having as a practical solution that removes the need to tune the baseline
\nlearning rate.\n<\/p>\n

<\/p>\n

\n

<\/a>Adapting to D<\/h4>\n

<\/p>\n

Conference awards often elevate papers that take interesting new approaches to
\nold problems, and this is certainly the case here. Our approach sounds like
\nit wouldn\u2019t work: we take an upper bound that would normally be used
\nto show convergence, and we invert it, to instead give a lower bound
\nd<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/math> on a key
\nquantity D<\/mi> =<\/mo> \u2225<\/mo>x<\/mi><\/mrow>0<\/mn><\/mrow><\/msub> \u2212<\/mo> x<\/mi><\/mrow>\u2217<\/mo><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow><\/math>.
\nSince D<\/mi><\/math> plays
\nthe key role of the numerator of the step size in convex Lipscthiz optimization, we can use
\nthis bound d<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/math>
\nto define a step-size: <\/p>\n\n\n
\n\n \u03b3<\/mi><\/mrow>k<\/mi><\/mrow><\/msub> =<\/mo> d<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/mrow> \n\u2211<\/mo>\n <\/mrow>i<\/mi><\/mrow>k<\/mi><\/mrow><\/munderover> \u2225<\/mo>g<\/mi><\/mrow>i<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow>2<\/mn><\/mrow><\/msup><\/mrow><\/msqrt><\/mrow><\/mfrac>.<\/mo>\n<\/math><\/td>\n<\/tr>\n<\/table>\n

<\/p>\n

Using this step-size in a plug-and-play style gives the D-Adaptation method.
\nSimple!\n<\/p>\n

<\/p>\n

The particular bound we use is a slight refinement of existing convergence
\nbounds: <\/p>\n\n\n
\n

\n\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>d<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub> (<\/mo>f<\/mi>(<\/mo>x<\/mi><\/mrow>k<\/mi><\/mrow><\/msub>)<\/mo> \u2212<\/mo> f<\/mi><\/mrow>\u2217<\/mo><\/mrow><\/msub><\/mrow>)<\/mo><\/mrow>\u2264<\/mo> D<\/mi> \u2225<\/mo>\u2211<\/mo>\n<\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>g<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow>+<\/mo>1<\/mn><\/mrow>\n2<\/mn><\/mrow><\/mfrac>\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>\u03b3<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub>d<\/mi><\/mrow>k<\/mi><\/mrow>2<\/mn><\/mrow><\/msubsup> \u2225<\/mo>g<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow>2<\/mn><\/mrow><\/msup>\u2212<\/mo>1<\/mn><\/mrow>\n2<\/mn><\/mrow><\/mfrac>\u03b3<\/mi><\/mrow>n<\/mi>+<\/mo>1<\/mn><\/mrow><\/msub> \u2225<\/mo>\u2211<\/mo>\n<\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>g<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow>2<\/mn><\/mrow><\/msup>,<\/mo>\n<\/math><\/td>\n<\/tr>\n<\/table>\n

<\/p>\n

and the rearrangement is simple. We first use that
\nf<\/mi>(<\/mo>x<\/mi><\/mrow>k<\/mi><\/mrow><\/msub>)<\/mo> \u2212<\/mo> f<\/mi><\/mrow>\u2217<\/mo><\/mrow><\/msub>\u2265<\/mo> 0<\/mn><\/math>, to
\ngive <\/p>\n\n\n
\n\n 0<\/mn> \u2264<\/mo> D<\/mi> \u2225<\/mo>\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>g<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> +<\/mo> 1<\/mn><\/mrow> \n2<\/mn><\/mrow><\/mfrac>\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>\u03b3<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub>d<\/mi><\/mrow>k<\/mi><\/mrow>2<\/mn><\/mrow><\/msubsup> \u2225<\/mo>g<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow>2<\/mn><\/mrow><\/msup> \u2212<\/mo>1<\/mn><\/mrow> \n2<\/mn><\/mrow><\/mfrac>\u03b3<\/mi><\/mrow>n<\/mi>+<\/mo>1<\/mn><\/mrow><\/msub> \u2225<\/mo>\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>g<\/mi><\/mrow>\nk<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow>2<\/mn><\/mrow><\/msup>,<\/mo>\n<\/math><\/td>\n<\/tr>\n<\/table>\n

<\/p>\n

Then pull D<\/mi><\/math>
\nover to the left: <\/p>\n\n\n
\n\n D<\/mi> \u2265<\/mo> d<\/mi><\/mrow>k<\/mi><\/mrow><\/msub> =<\/mo> \u03b3<\/mi><\/mrow>n<\/mi>+<\/mo>1<\/mn><\/mrow><\/msub> \u2225<\/mo>\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>g<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow>2<\/mn><\/mrow><\/msup> \u2212<\/mo>\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>\u03b3<\/mi><\/mrow>k<\/mi><\/mrow><\/msub>d<\/mi><\/mrow>k<\/mi><\/mrow>2<\/mn><\/mrow><\/msubsup> \u2225<\/mo>g<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow>2<\/mn><\/mrow><\/msup><\/mrow> \n 2<\/mn> \u2225<\/mo>\u2211<\/mo>\n <\/mrow>k<\/mi>=<\/mo>0<\/mn><\/mrow>n<\/mi><\/mrow><\/munderover>g<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/mrow>\u2225<\/mo><\/mrow> <\/mrow><\/mfrac> .<\/mo>\n<\/math><\/td>\n<\/tr>\n<\/table>\n

<\/p>\n

Why does this look crazy? Well for one thing, we have no reason to expect
\nd<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/math>
\nto be positive, in fact, it usually isn\u2019t. The solution to the positivity issue<\/p>\n

turns out to be simple. Instead of using the bound directly, we define
\nd<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/math> to be
\nthe max over the bounds over all previous steps. To \u201cbootstrap\u201d the process, we start
\nwith d<\/mi><\/mrow>0<\/mn><\/mrow><\/msub> =<\/mo> 1<\/mn>0<\/mn><\/mrow>\u2212<\/mo>8<\/mn><\/mrow><\/msup><\/math>
\nor a similar small positive value. Given the seeming looseness of this
\napproach, we wouldn\u2019t expect this lower bound to provide a close estimate to
\nD<\/mi><\/math>, but in fact it does!
\nIn practice, the d<\/mi><\/math>
\nsequence grows exponentially, from arbitrarily small initial
\nd<\/mi><\/mrow>0<\/mn><\/mrow><\/msub><\/math> to
\nvalues as good as hand tuned choices. We are even able to show that in theory, it
\nasymptotically approaches the true D to within a factor of 0.366.\n<\/p>\n

<\/p>\n

\n

<\/a>Convergence properties<\/h3>\n

<\/p>\n

\n

\n <\/p>\n

\u201cNon-asymptotic results tell you \u2019what\u2019s really going on\u2019, but
\n asymptotics tell you what\u2019s *really* going on\u201d – Sam Power<\/a><\/p>\n<\/blockquote>\n

<\/p>\n

Our key theory result is simple: we show that for a sufficiently large number of training
\nsteps n<\/mi><\/math>,
\nour approach yields convergence as fast as if you had of set the step size using
\nknowledge of the true D from the beginning: <\/p>\n\n\n
\n\n f<\/mi>(<\/mo>x<\/mi><\/mrow>^<\/mo><\/mover><\/mrow>n<\/mi><\/mrow><\/msub>)<\/mo> \u2212<\/mo> f<\/mi>(<\/mo>x<\/mi><\/mrow>\u2217<\/mo><\/mrow><\/msub>)<\/mo> =<\/mo> <\/mi> (<\/mo> DG<\/mi><\/mrow>\nn<\/mi> +<\/mo> 1<\/mn><\/mrow><\/msqrt><\/mrow><\/mfrac> <\/mrow>)<\/mo><\/mrow>,<\/mo>\n<\/math><\/td>\n<\/tr>\n<\/table>\n

<\/p>\n

where x<\/mi><\/mrow>^<\/mo><\/mover><\/mrow>n<\/mi><\/mrow><\/msub><\/math> is the
\naverage iterate, and G<\/mi><\/math>
\nis a bound on the gradient norms. This is an interesting result for a
\nnumber of reasons. Firstly, it\u2019s the first such \u201casymptotic\u201d result for this<\/p>\n

problem class to have no additional log factors – you pay no penalty for
\nnot knowing D! Other methods for the problem class we consider: convex,
\nnon-smooth Lipschitz functions, require knowledge of other unknowns, such as
\nf<\/mi>(<\/mo>x<\/mi><\/mrow>\u2217<\/mo><\/mrow><\/msub>)<\/mo><\/math> or
\nrequire line searches, back-tracking or other complexities. A chief advantage of our
\nmethod is it\u2019s so damn simple.\n<\/p>\n

<\/p>\n

The major down-side of this asymptotic result is that we
\ndon\u2019t know how many optimization steps need to be taken,
\nn<\/mi><\/math>, before
\nthis asymptotic behavior kicks in. Fortunately, in practice, it kicks in after less than a
\nfew hundred steps, typically 1-2% of the total training time at most. This corresponds to
\nwhen the d<\/mi><\/mrow>k<\/mi><\/mrow><\/msub><\/math>
\nestimate becomes within a constant factor of the true
\nD<\/mi><\/math>. We
\nsee this across more than 20 test problems, across deep-learning problems as well as
\nsimpler logistic regression problems.\n<\/p>\n

<\/p>\n

The asymptotic bound we prove appears to be the true indictor of the behavior of
\nour method in practice, although we give a non-asymptotic bound as well, which shows
\nthat the convergence is never worse than a log factor off the best-possible rate, even
\nwhen n<\/mi><\/math>
\nis small. Approaches from the \u201cparameter-free\u201d literature such as coin-betting give
\nslightly better non-asymptotic rates for this class (sqrt(log) factor difference),
\nhowever we have developed an improved variant of D-Adaptation that matches the
\nnon-asymptotic rates of those parameter-free methods, known as Prodigy
\n(https:\/\/arxiv.org\/pdf\/2306.06101.pdf<\/a>). Prodigy seems to be an overall improvement
\nover D-Adaptation in practice also, so we recommend its use as the primary variant
\nof the method.\n<\/p>\n

<\/p>\n

\n

<\/a>More on non-asymptotics<\/h4>\n

<\/p>\n

The non-asymptotic rate given by approaches such as coin betting for the average
\niterate is: <\/p>\n\n\n
\n

\n f<\/mi>(<\/mo>x<\/mi><\/mrow>^<\/mo><\/mover><\/mrow>n<\/mi><\/mrow><\/msub>)<\/mo> \u2212<\/mo> f<\/mi>(<\/mo>x<\/mi><\/mrow>\u2217<\/mo><\/mrow><\/msub>)<\/mo> =<\/mo> <\/mi> (<\/mo>DG<\/mi>log<\/mi> \u2061 <\/mo> (<\/mo>1<\/mn> +<\/mo> D<\/mi>\u2215<\/mo>d<\/mi><\/mrow>0<\/mn> <\/mrow> <\/msub><\/mrow>)<\/mo><\/mrow><\/mrow><\/msqrt><\/mrow>\n n<\/mi> +<\/mo> 1<\/mn><\/mrow><\/msqrt><\/mrow><\/mfrac> <\/mrow>)<\/mo><\/mrow>.<\/mo>\n<\/math><\/td>\n<\/tr>\n<\/table>\n

<\/p>\n

Essentially you pay a log<\/mi> \u2061 <\/mo> (<\/mo>1<\/mn> +<\/mo> D<\/mi>\u2215<\/mo>d<\/mi><\/mrow>0<\/mn> <\/mrow> <\/msub> )<\/mo><\/mrow><\/msqrt><\/math> penalty
\nfor not having knowledge of D<\/mi><\/math>
\nbefore-hand. There is a much simpler approach that gives essentially the same rate
\nfor this complexity class: Grid Search! In fact, the simple approach of running
\nR<\/mi> =<\/mo> log<\/mi> \u2061 <\/mo>(<\/mo>D<\/mi><\/mrow>max<\/mi> \u2061 <\/mo><\/mrow><\/msub>\u2215<\/mo>D<\/mi><\/mrow>min<\/mi> \u2061 <\/mo><\/mrow><\/msub>)<\/mo><\/math> runs with a log-spaced
\ngrid of D values (i.e. 1<\/mn>0<\/mn><\/mrow>\u2212<\/mo>8<\/mn><\/mrow><\/msup>,<\/mo>1<\/mn>0<\/mn><\/mrow>\u2212<\/mo>7<\/mn><\/mrow><\/msup>,<\/mo>1<\/mn>0<\/mn><\/mrow>\u2212<\/mo>6<\/mn><\/mrow><\/msup>,<\/mo>\u2026<\/mi> \u2061<\/mo><\/mspace><\/math>)
\nbracketing the true D. Since our total step-budget is
\nn<\/mi> +<\/mo> 1<\/mn><\/math>,
\neach run must be of length (n+1)\/R, and so taking the best
\nx<\/mi><\/mrow>^<\/mo><\/mover><\/math> by
\nfunction value gives a bound:\n<\/p>\n

\n f<\/mi>(<\/mo>x<\/mi><\/mrow>^<\/mo><\/mover><\/mrow>n<\/mi><\/mrow><\/msub>)<\/mo> \u2212<\/mo> f<\/mi>(<\/mo>x<\/mi><\/mrow>\u2217<\/mo><\/mrow><\/msub>)<\/mo><\/mtd> =<\/mo> <\/mi> (<\/mo> DG<\/mi><\/mrow>\n(<\/mo>n<\/mi> +<\/mo> 1<\/mn>)<\/mo>\u2215<\/mo>R<\/mi><\/mrow><\/msqrt><\/mrow><\/mfrac> <\/mrow>)<\/mo><\/mrow><\/mspace><\/mtd> <\/mtd> \n <\/mspace><\/mtd><\/mtr><\/mtd> =<\/mo> <\/mi> (<\/mo>DG<\/mi>R<\/mi><\/mrow><\/msqrt><\/mrow>\nn<\/mi> +<\/mo> 1<\/mn><\/mrow><\/msqrt><\/mrow><\/mfrac> <\/mrow>)<\/mo><\/mrow><\/mspace><\/mtd> <\/mtd> \n <\/mspace><\/mtd><\/mtr><\/mtd> =<\/mo> <\/mi> (<\/mo>DG<\/mi>log<\/mi> \u2061 <\/mo> (<\/mo>D<\/mi><\/mrow>max<\/mi> \u2061 <\/mo> <\/mrow> <\/msub> \u2215<\/mo>D<\/mi><\/mrow>min<\/mi> \u2061 <\/mo> <\/mrow> <\/msub><\/mrow>)<\/mo><\/mrow><\/mrow><\/msqrt><\/mrow>\n n<\/mi> +<\/mo> 1<\/mn><\/mrow><\/msqrt><\/mrow><\/mfrac> <\/mrow>)<\/mo><\/mrow>.<\/mo><\/mspace><\/mtd> <\/mtd> \n <\/mspace><\/mtd><\/mtr><\/mtable><\/math>
\n<\/p>\n

Often in computer-science research we ignore log factors as unimportant, or
\n\u201cnot-real\u201d, but in this case, the log factor is very real! It corresponds to the
\ncost of running multiple training runs and throwing away all but the best
\nrun.<\/p>\n

<\/p>\n

Given this is the same sqrt(log) dependence as for coin betting, what
\ndifferentiates parameter-free approaches such as coin-betting? They are developed for
\na much more general class of problems, Online Linear Optimization (OLO),
\nand so when applied to convex non-smooth optimization as we do here,
\nthey are not able to make use of the additional problem structure. Note
\nthat the grid search method above obviously can\u2019t achieve a sqrt-log-factor
\nfree rate asymptotically, since you have to actually perform each of the
\nR<\/mi><\/math>
\ntraining runs. It\u2019s unclear if coin-betting approaches can avoid the log factor
\nasymptotically, this is an interesting direction for future research. Lower bounds are
\nknown for the online linear optimization setting that show that any non-asymptotic
\nrate must include this sqrt-log factor.\n<\/p>\n

<\/p>\n

\n

<\/a>Why previous parameter-free methods don\u2019t work as well<\/h3>\n

<\/p>\n

A major component of our work was the effort it took to go from a theoretically nice
\nalgorithm to a practical method that works on deep learning problems. We
\nhad to identify why other learning-rate free methods fail, and we had to
\nadapt our method to work on top of modern optimization methods such as
\nAdam.\n<\/p>\n

<\/p>\n

The key insight turned out to be SCHEDULING. By scheduling, I mean a
\nsequence of constants that multiply the learning rate, that includes potentially
\nwarm-up at the beginning of training, and annealing of the learning rate towards the
\nend of training.\n<\/p>\n

<\/p>\n

Using theoretically motivated schedules such as
\n1<\/mn>\u2215<\/mo>k<\/mi> +<\/mo> 1<\/mn><\/mrow><\/msqrt><\/math>
\nlearning rate decay turn out to work terribly in practice almost across the board on
\ndeep learning problems. Existing coin-betting approaches prior to our work all relied
\non very poorly performing schedules. This is why the COCOB method we tested as a
\nbaseline under-performed The schedule is an implicit part of the method, arising
\ndirectly from the mathematical formulation, and so its difficult to apply an arbitrary
\nschedule on top of it, in contrast, D-Adaptation natively supports the use of arbitrary
\nschedules.\n<\/p>\n

<\/p>\n

Using existing empirically motivated schedules, such as those already widely in
\nuse in deep learning practice, leads to huge improvements in performance. In
\ncollaboration with Ashok and Harsh, I have also been working on incorporating
\nschedule support into practical parameter-free methods, giving a method we call
\nMechanic (https:\/\/arxiv.org\/abs\/2306.00144<\/a>).\n<\/p>\n

<\/p>\n

\n

<\/a>Practical performance of D-Adaptation<\/h3>\n

<\/p>\n

Our Adam variant of D-Adaptation is a drop-in replacement for the PyTorch Adam
\nclass, requiring no other changes to the code-base. We tested across a large selection
\nof modern deep learning problems, including auto-regressive language models,
\nimage-classification, image-2-image tasks, object recognition, recommendation
\nsystems, masked-language models and more. In fact, we performed the largest
\ncomparison performed for any optimization paper for a new method, outside of 10+
\nauthor papers from Google.\n<\/p>\n

<\/p>\n

D-Adaptation matched the final test performance of laboriously hand-tuned
\nlearning rates on 9 of 10 deep learning problems, and all 12 logistic regression
\nproblems tested. We have also received overwhelming feedback from the ML
\ncommunity that the method \u201cjust-works\u201d and is a true viable replacement for
\nhand-tuned learning rates.\n<\/p>\n

<\/p>\n

ICML is not a theoretically focused venue; reviewers highly value papers
\nwhose contribution have clear practical impact together with strong theory.
\nBefore D-Adaptation, there were no methods that could be \u201cdropped-in\u201d to
\na code base without other changes to provide adaptivity to the learning
\nrate that actually worked well in practice. The ease of use of our method
\nand its clear practical utility were recognized by our reviewers, with final
\nreview scores of 9, 8 and 6. We were fortunate to have highly communicative
\nreviewers, who suggested improvements, pointed out errors (even reviewing the
\nappendix carefully!) and overall greatly improved our work. A triumph of peer
\nreview!\n<\/p>\n

<\/p>\n

\n

<\/a>Limitations<\/h3>\n

<\/p>\n

D-Adapt Adam uses an additional memory buffer on top of regular Adam, and so it
\nhas overall higher memory use. It can also fail on problems where training is very
\nunstable. We experienced issues with it\u2019s convergence on ViT (vision transformer)
\ntraining for example.\n<\/p>\n

<\/p>\n

Since setting the learning rate globally with D-Adaptation requires aggregating
\ndot products and norms from across all parameters, it will requiring syncing
\ninformation across nodes when using sharded data-parallel training, which may have
\na performance hit. We found that applying it separately at the block or node level
\ndidn\u2019t work well, the global syncing appears to be necessary.\n<\/p>\n

<\/p>\n

Our theory also has limitations. It only applies in the deterministic setting, which
\nis much more limited than the stochastic or online settings that other methods such
\nas coin-betting or DoG have theory for. Given the method seems to work well for
\nstochastic problems, I see developing theory for this case as an interesting research
\nproblem.\n<\/p>\n

<\/p>\n

D-Adaptation MUST be used with a schedule. I have seen several comparisons
\nagainst D-Adaptation that don\u2019t mention the use of schedules, and likely omitted
\nthem. This is crucial for the performance of the method. We recommend the use of<\/p>\n

the linear decay schedule with potentially a warmup (5%?) if warmup is customary
\nfor the problem being tackled.\n<\/p>\n

<\/p>\n

\n

<\/a>What\u2019s Next<\/h3>\n

<\/p>\n

It\u2019s rather annoying that we have to choose a schedule to use with D-Adaptation….
\nsurely there is some way to determine data-adaptive schedules automatically? Stay
\ntuned for a solution! <\/p>\n","protected":false},"excerpt":{"rendered":"

Konstantin and I were honored with an ICML Outstanding paper award for our paper \u201cLearning-Rate-Free Learning by D-Adaptation\u201d. Besides the award, our implementation has already been downloaded nearly a million times over the last year (most before the award), and is seeing wide-spread use in training and fine-tuning deep learning models, particularly LoRAs. Read on … Continue reading D-Adaptatation Wins an Outstanding Paper Award at ICML 2023!<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts\/127"}],"collection":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=127"}],"version-history":[{"count":6,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts\/127\/revisions"}],"predecessor-version":[{"id":133,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts\/127\/revisions\/133"}],"wp:attachment":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=127"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=127"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=127"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}